Blog Post

Azure Synapse Analytics Blog
5 MIN READ

Metadata-Based Ingestion in Synapse with Delta Lake

madhuvigupta's avatar
madhuvigupta
Icon for Microsoft rankMicrosoft
Jul 19, 2023
  Overview     The crucial first step in any ETL (extract, transform, load) process or data engineering program is ingestion, which involves dealing with multiple data sources and entities or d...
Updated Jul 11, 2023
Version 1.0
Synapse Pipelines
Synapse SQL
madhuvigupta's avatar
madhuvigupta
Icon for Microsoft rankMicrosoft
Oct 09, 2023

NeoDunn Thank you for reaching out. Please find below explanation of the cell 3. Hope this is helpful.

 

val createDeltaLoadTasks = Future.sequence( allEntities.map( k => mergeDeltaRawToGold(k._1, k._2, k._4, k._5, k._7)) )

 

  • allEntities is an iterable (likely a collection or array) containing tuples of data (k._1, k._2, k._4, k._5, k._7).
  • The map function is applied to each tuple in allEntities, invoking the mergeDeltaRawToGold method with specific parameters from each tuple.
  • mergeDeltaRawToGold is expected to return a Future[Boolean].
  • Future.sequence() transforms the collection of Future[Boolean] into a single
  • Future[Seq[Boolean]], representing the completion status of each asynchronous

The mergeFullRawToGold method is designed to operate as a Future, utilizing asynchronous processing. It accepts a metadata column as an input parameter and returns a boolean value encapsulated within a Future. Concurrent execution is achieved using Future.sequence(), allowing for the parallel execution of mergeFullRawToGold for multiple metadata entries stored in an ArrayBuffer.

In addition, the Await.result function is used to block the main thread and await the result of a given Future within a defined duration. If the result is not ready within the specified time or if it completes with a failure, Await.result will throw an exception, effectively handling the outcome of the asynchronous operation.