Forum Discussion
how to parse Multiline log from the files and ingest into Azure Sentinel
Team, Can any one let me know how to parse multiline logs that are generated from various applications by using the custom application log method.
Attached is the logs where it is getting split into multiple lines where this is a single event when it gets generated in an application
That screen shot helps, but you cant see the Table or Column names to be sure.
Using SigninLogs table as an example and the column DeviceDetail
You may just be able to pick a row:
SigninLogs
| where DeviceDetail has "Rich Client"
| project DeviceDetail.browser
| DeviceDetail_browser |
|---|
| Rich Client 5.2.2.0 |
| Rich Client 5.2.2.0 |
| Rich Client 5.2.2.0 |
Other data sources may need parse_json / mv-expand or an example here
- Thanks Clive for the response. Custom logs can be ingested into sentinel either through single entry per line or using a timestamp matching as per the documentation. Since my logs do not follow either one of the time formats as given below, i had to use the new entry per line.
YYYY-MM-DD HH:MM:SS
M/D/YYYY HH:MM:SS AM/PM
Mon DD, YYYY HH:MM:SS
yyMMdd HH:mm:ss
ddMMyy HH:mm:ss
MMM d hh:mm:ss
dd/MMM/yyyy:HH:mm:ss zzz
yyyy-MM-ddTHH:mm:ssK
Below is the sample event that has been ingested as provided earlier in the screenshot. Any new line in the event is being considered as a new event as i had the only option to select since the time format of the custom logs ingestion method varies. All the events start with the time fields. the second event also starts with time fields but the event is spread across multiplelines. Since we have selected new line per entry the agent is picking up each and every new line as new event.
Kindly suggest if there are any other ways to ingest these logs
2021/09/18 23:26:17.091 [P122]: (polo.exe__0) (10008) handling (GetServerInfo)
2021/09/18 23:26:17.091 [P122]: (polo.exe__0) (10008) handled (GetServerInfo) [0.000s] [40748Kb] [Peak 58660Kb]
System previously shut down with the following event actions in progress or pending.
All event actions have been reset.
In Progress, Event ID = 151663, Rule ID = 5200400, Last Action Time = 2021/09/ 1 15:38:05, Affected Symbols:
ACCOUNTS: AFPARAM9
TIMEPER: DIM1SET
ENTITIES: DIM2SET, APG_LEGL, BE_, BEPGJ, CNCTC, CNDAT, CNTXC, CNZTC, CNTDS, CNTHB, CNTPH, CNHHS, CNDCB, CN_, CNTAS, CNTIS, CNPES, CNAPG, HKPOG, COABB, CO_, DEPUC, DE_, DEPGG, DECAS, FR_, FRAPG, GBVUL, GB_, GBMIN, CHSWV, HOL_, CHAMH, CHTBC, NLPGH, CHTEC, CHPFI, AEPIM, CHPPG, KR_, KRAPG, SEABB, SE_, SECYJ, SG_, SGPGH, USKKM, USKEC, USVEN, US_, USGCS, USGHV, USPAL, USPBL, USPGO, USPJC, USPSB, USPSL, NPPIO, USPIO, VNTRA, VN_, AE_, AEAPG, AEFPG, AEDPG, AR_, ARPGA, AT_, ATPGA, ATPGS, AU_, AUMIN, AUMIP, AUPGH, BG_, BGAPG, BHARE, BH_, BRABB, BR_, CA_, CAAPG, CHSEC, CHSEM, CHTRF, CH_, CHPGS, CHCAS, CL_, CLAPG, CZ_, CZAPG, DK_, DKAPG, EE_, EEAPG, EGTRA, EG_, EGHPG, EGPPG, ES_, ESPGH, FI_, FIAPG, GR_, GRAPG, HR_, HRAPG, HU_, HUAPG, ID_, IDPGH, IE_, IEAPG, IL_, ILPGA, IN_, INAPG, INAPT, ITIMG, IT_, ITPGI, JONEA, JO_, JOAPG, JP_, JPAPG, KWABB, KW_, MX_, MXPGA, MY_, MYMIN, MYPGH, NLVEN, NL_, NLPGJ, NO_, NOPGN, NZ_, NZPGL, OM_, OMAPG, PAABB, PASUC, PA_, PE_, PEAPG, PH_, PHMIN, PHPOG, PK_, PKAPG, PL_, PLAPG, PLPGS, PTABB, PT_, QA_, QAAPG, ROABB, RO_, RUREL, RU_, RUAPG, SAACE, SA_, SK_, SKAPG, SYS_, ZGRS, ZEUR, ZUSD, ZCHF, ZIGE, ZALB, ZALC, ZALG, ZALS, ZTAED, ZTAUD, ZTBRL, ZTCAD, ZTCHF, ZTCNY, ZTCZK, ZTDKK, ZTEUR, ZTGBP, ZTINR, ZTJPY, ZTKRW, ZTMXN, ZTNOK, ZTPLN, ZTRUB, ZTSAR, ZTSEK, ZTSGD, ZTTHB, ZTTRY, ZTUSD, ZTZAR, THABB, THAPH, THKEM, TH_, TRITE, TR_, TRPGE, TRCMC, TW_, TWAPG, UA_, UAAPG, ZAVSE, ZA_, ZAMIN, ZAAPG, IR_, IRABB, IQ_, IQAPG
DETAILS: DIM3SET
CURRENCY: DIM4SET
SEGMENTS: DIM5SET
ELEMENTS: DIM6SET
CONTROLS: DEFTAXSTP12
In Progress, Event ID = 151706, Rule ID = 9000890, Last Action Time = 2021/09/ 1 15:42:04, Affected Symbols:
ACCOUNTS: Val_Allow_TaxPctCalc
TIMEPER: A2109YTD
ENTITIES: CNCTC, CNDAT, CNTXC, CNZTC, CNTDS, CNTHB, CNTPH, CNHHS, CNDCB, CNTAS, CNTIS, CNPES, CNAPG, HKPOG, DEPUC, DEPGG, FRAPG, GBVUL, CHAMH, CHTBC, CHTEC, CHPPG, SEABB, USKKM, USVEN, ATPGA, AUMIN, BRABB, CAAPG, CHSEC, CHSEM, CHTRF, CZAPG, EGTRA, FIAPG, IDPGH, INAPG, INAPT, ITPGI, JPAPG, MXPGA, MYPGH, NOPGN, NZPGL, PKAPG, PLAPG, PLPGS, RUREL, RUAPG, SAACE, THABB, TRPGE, TWAPG, ZAAPG
DETAILS: DIM3SET
CURRENCY: DIM4SET
SEGMENTS: DIM5SET
ELEMENTS: GrsCY_IAS
CONTROLS: DIM7SET