how to parse Multiline log from the files and ingest into Azure Sentinel

Question

Team,&nbsp;Can any one let me know how to parse multiline logs that are generated from various applications by using the custom application log method.

m_zorich · Answer

Could you share an example log and what fields you wanted to extract? There are a number of ways to parse data in KQL

pavankemi · Answer

m_zorich&nbsp;Attached is the logs where it is getting split into multiple lines where this is a single event when it gets generated in an application&nbsp;

clivewatson · Answer

pavankemi&nbsp;
That screen shot helps, but you cant see the Table or Column names to be sure.
Using SigninLogs table as an example and the column DeviceDetail

You may just be able to pick a row:
SigninLogs
| where DeviceDetail has "Rich Client"
| project DeviceDetail.browser
&nbsp;

DeviceDetail_browser

Rich Client 5.2.2.0

Other data sources may need parse_json / mv-expand or an example here
&nbsp;

pavankemi · Answer

Thanks Clive for the response. Custom logs can be ingested into sentinel either through single entry per line or using a timestamp matching as per the documentation. Since my logs do not follow either one of the time formats as given below, i had to use the new entry per line.YYYY-MM-DD HH:MM:SSM/D/YYYY HH:MM:SS AM/PMMon DD, YYYY HH:MM:SSyyMMdd HH:mm:ssddMMyy HH:mm:ssMMM d hh:mm:ssdd/MMM/yyyy:HH:mm:ss zzzyyyy-MM-ddTHH:mm:ssKBelow is the sample event that has been ingested as provided earlier in the screenshot. Any new line in the event is being considered as a new event as i had the only option to select since the time format of the custom logs ingestion method varies. All the events start with the time fields. the second event also starts with time fields but the event is spread across multiplelines. Since we have selected new line per entry the agent is picking up each and every new line as new event.Kindly suggest if there are any other ways to ingest these logs2021/09/18 23:26:17.091 [P122]: (polo.exe__0) (10008) handling (GetServerInfo)2021/09/18 23:26:17.091 [P122]: (polo.exe__0) (10008) handled (GetServerInfo) [0.000s] [40748Kb] [Peak 58660Kb]System previously shut down with the following event actions in progress or pending.All event actions have been reset.In Progress, Event ID = 151663, Rule ID = 5200400, Last Action Time = 2021/09/ 1 15:38:05, Affected Symbols:ACCOUNTS: AFPARAM9TIMEPER: DIM1SETENTITIES: DIM2SET, APG_LEGL, BE_, BEPGJ, CNCTC, CNDAT, CNTXC, CNZTC, CNTDS, CNTHB, CNTPH, CNHHS, CNDCB, CN_, CNTAS, CNTIS, CNPES, CNAPG, HKPOG, COABB, CO_, DEPUC, DE_, DEPGG, DECAS, FR_, FRAPG, GBVUL, GB_, GBMIN, CHSWV, HOL_, CHAMH, CHTBC, NLPGH, CHTEC, CHPFI, AEPIM, CHPPG, KR_, KRAPG, SEABB, SE_, SECYJ, SG_, SGPGH, USKKM, USKEC, USVEN, US_, USGCS, USGHV, USPAL, USPBL, USPGO, USPJC, USPSB, USPSL, NPPIO, USPIO, VNTRA, VN_, AE_, AEAPG, AEFPG, AEDPG, AR_, ARPGA, AT_, ATPGA, ATPGS, AU_, AUMIN, AUMIP, AUPGH, BG_, BGAPG, BHARE, BH_, BRABB, BR_, CA_, CAAPG, CHSEC, CHSEM, CHTRF, CH_, CHPGS, CHCAS, CL_, CLAPG, CZ_, CZAPG, DK_, DKAPG, EE_, EEAPG, EGTRA, EG_, EGHPG, EGPPG, ES_, ESPGH, FI_, FIAPG, GR_, GRAPG, HR_, HRAPG, HU_, HUAPG, ID_, IDPGH, IE_, IEAPG, IL_, ILPGA, IN_, INAPG, INAPT, ITIMG, IT_, ITPGI, JONEA, JO_, JOAPG, JP_, JPAPG, KWABB, KW_, MX_, MXPGA, MY_, MYMIN, MYPGH, NLVEN, NL_, NLPGJ, NO_, NOPGN, NZ_, NZPGL, OM_, OMAPG, PAABB, PASUC, PA_, PE_, PEAPG, PH_, PHMIN, PHPOG, PK_, PKAPG, PL_, PLAPG, PLPGS, PTABB, PT_, QA_, QAAPG, ROABB, RO_, RUREL, RU_, RUAPG, SAACE, SA_, SK_, SKAPG, SYS_, ZGRS, ZEUR, ZUSD, ZCHF, ZIGE, ZALB, ZALC, ZALG, ZALS, ZTAED, ZTAUD, ZTBRL, ZTCAD, ZTCHF, ZTCNY, ZTCZK, ZTDKK, ZTEUR, ZTGBP, ZTINR, ZTJPY, ZTKRW, ZTMXN, ZTNOK, ZTPLN, ZTRUB, ZTSAR, ZTSEK, ZTSGD, ZTTHB, ZTTRY, ZTUSD, ZTZAR, THABB, THAPH, THKEM, TH_, TRITE, TR_, TRPGE, TRCMC, TW_, TWAPG, UA_, UAAPG, ZAVSE, ZA_, ZAMIN, ZAAPG, IR_, IRABB, IQ_, IQAPGDETAILS: DIM3SETCURRENCY: DIM4SETSEGMENTS: DIM5SETELEMENTS: DIM6SETCONTROLS: DEFTAXSTP12In Progress, Event ID = 151706, Rule ID = 9000890, Last Action Time = 2021/09/ 1 15:42:04, Affected Symbols:ACCOUNTS: Val_Allow_TaxPctCalcTIMEPER: A2109YTDENTITIES: CNCTC, CNDAT, CNTXC, CNZTC, CNTDS, CNTHB, CNTPH, CNHHS, CNDCB, CNTAS, CNTIS, CNPES, CNAPG, HKPOG, DEPUC, DEPGG, FRAPG, GBVUL, CHAMH, CHTBC, CHTEC, CHPPG, SEABB, USKKM, USVEN, ATPGA, AUMIN, BRABB, CAAPG, CHSEC, CHSEM, CHTRF, CZAPG, EGTRA, FIAPG, IDPGH, INAPG, INAPT, ITPGI, JPAPG, MXPGA, MYPGH, NOPGN, NZPGL, PKAPG, PLAPG, PLPGS, RUREL, RUAPG, SAACE, THABB, TRPGE, TWAPG, ZAAPGDETAILS: DIM3SETCURRENCY: DIM4SETSEGMENTS: DIM5SETELEMENTS: GrsCY_IASCONTROLS: DIM7SET

Forum Discussion

how to parse Multiline log from the files and ingest into Azure Sentinel

4 Replies

Resources