Custom Log schema design

%3CLINGO-SUB%20id%3D%22lingo-sub-3417478%22%20slang%3D%22en-US%22%3ECustom%20Log%20schema%20design%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3417478%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20retrieving%20sign%20in%20and%20activity%20audit%20data%20from%203%20source%20systems%20with%203%20different%20scripts%2C%20one%20for%20each%20system%2C%20and%20preparing%20to%20send%20them%20to%20a%20custom%20log%20in%20Azure%20Monitor%2FLog%20Analytics%2FSentinel.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20know%20that%20once%20in%20Azure%20Monitor%20I%20will%20be%20writing%20queries%20and%20having%20to%20join%20or%20union%20the%203%20datasets%2C%20so%20my%20questions%20are%3B%3C%2FP%3E%3CUL%3E%3CLI%3EShould%20I%20try%20normalise%20the%20fields%2C%20and%20add%20all%20three%20logs%20into%20a%20single%20Azure%20Monitor%20log%20table%3F%3C%2FLI%3E%3CLI%3EShould%20I%20keep%20them%20in%203%20separate%20tables%20and%20use%20%22join%22%20commands%20to%20bring%20them%20together%3F%3C%2FLI%3E%3C%2FUL%3E%3CP%3EI%20remember%20in%20Filebeat%2C%20it%20would%20hold%20many%20different%20log%20sources%20in%20the%20one%20index%20(table)%2C%20and%20so%20wondering%20if%20i%20should%20do%20the%20same%20here%20in%20Azure%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-3417478%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EKusto%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Data%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-3420057%22%20slang%3D%22en-US%22%3ERe%3A%20Custom%20Log%20schema%20design%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3420057%22%20slang%3D%22en-US%22%3EOne%20other%20option%20is%20to%20mimic%20what%20MS%20does%20with%20the%20Advanced%20Security%20Information%20Model%20(ASIM)%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fnormalization%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fnormalization%3C%2FA%3E%3CBR%20%2F%3EIngest%20the%20data%20as%20is%20and%20write%20a%20KQL%20function%20that%20does%20the%20normalization%20which%20you%20then%20use%20in%20your%20queries.%20Saves%20having%20everyone%20having%20to%20memorize%20the%20different%20layouts%20of%20data%3C%2FLINGO-BODY%3E
Frequent Contributor

I am retrieving sign in and activity audit data from 3 source systems with 3 different scripts, one for each system, and preparing to send them to a custom log in Azure Monitor/Log Analytics/Sentinel.

 

I know that once in Azure Monitor I will be writing queries and having to join or union the 3 datasets, so my questions are;

  • Should I try normalise the fields, and add all three logs into a single Azure Monitor log table?
  • Should I keep them in 3 separate tables and use "join" commands to bring them together?

I remember in Filebeat, it would hold many different log sources in the one index (table), and so wondering if i should do the same here in Azure?

1 Reply
One other option is to mimic what MS does with the Advanced Security Information Model (ASIM) https://docs.microsoft.com/en-us/azure/sentinel/normalization
Ingest the data as is and write a KQL function that does the normalization which you then use in your queries. Saves having everyone having to memorize the different layouts of data