Forum Discussion
DiskoSuperStar
Dec 02, 2024Copper Contributor
Clarification on Staging Directory Usage for SAP CDC Connector in Azure Data Factory
Hi!
I'm currently working on a project where we are ingesting data from SAP using the SAP CDC connector in Azure Data Factory(Data flow). The source is S4HAHA CDS views.
We are using a staging directory for the data flow with a checkpoint mechanism, similar to described here:
https://learn.microsoft.com/en-us/azure/data-factory/connector-sap-change-data-capture
My question is:
- Does the staging directory only act as a temporary storage location during ingestion from sap? If i understand correctly its used for retries, but no real usage once the deltas have been ingested.
- After the data has been loaded to the destination(in our case container inside of ADLS), is the data needed for maintaining delta states?
- Can the data be safely deleted(from the staging container) without impacting the subsequent load runs?
We were thinking of implementing a 7 day retention policy on the staging container so we can manage storage efficiently.
Thank you in advance for any information regarding this.
No RepliesBe the first to reply