Advice architecture, streaming data

%3CLINGO-SUB%20id%3D%22lingo-sub-2468382%22%20slang%3D%22en-US%22%3EAdvice%20architecture%2C%20streaming%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2468382%22%20slang%3D%22en-US%22%3E%3CDIV%20class%3D%22p-rich_text_section%22%3EI%E2%80%99m%20looking%20for%20advise%20with%20pros%20%2F%20cons%20on%20the%20following%3A%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%3CSTRONG%3Edata%20%26gt%3B%20azure%20event%20hub%20%26gt%3B%20databricks%20structured%20streaming%20%26gt%3B%20delta%20lake%20(bronze)%3C%2FSTRONG%3E%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3Evs%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%20class%3D%22p-rich_text_section%22%3E%3CSTRONG%3Edata%20%26gt%3B%20azure%20event%20hub%20%26gt%3B%20event%20hub%20capture%20to%20Azure%20Data%20Lake%20gen%202%20%26gt%3B%20Databricks%20Autoloader%20%26gt%3B%20delta%20lake(bronze)%3C%2FSTRONG%3E%3C%2FDIV%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAzure%20event%20hub%20has%20only%207%20day%20retention.%3C%2FP%3E%3CP%3EThe%20need%20is%20not%20on%20realtime%20but%20more%20batch%20with%20trigger%20once%20and%20schedule%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor
I’m looking for advise with pros / cons on the following:
 
 
data > azure event hub > databricks structured streaming > delta lake (bronze)
 
vs
 
data > azure event hub > event hub capture to Azure Data Lake gen 2 > Databricks Autoloader > delta lake(bronze)

 

 

Azure event hub has only 7 day retention.

The need is not on realtime but more batch with trigger once and schedule

0 Replies