SOLVED
Home

Azure not collecting custom log data

%3CLINGO-SUB%20id%3D%22lingo-sub-631780%22%20slang%3D%22en-US%22%3EAzure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-631780%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20creating%20a%20custom%20log%20on%20my%20windows%20VM%20and%20would%20like%20to%20query%20it%20on%20monitor%20log.%3C%2FP%3E%3CP%3EI've%20set%20up%20my%20custom%20log%20files%20just%20as%20Microsoft%20tell%20us%20to%20do%20on%20this%20archive%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-sources-custom-logs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-sources-custom-logs%3C%2FA%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eall%20went%20well%20and%20my%20custom%20log%20are%20there%20in%20the%20log%20analytics%20schema%2C%3C%2FP%3E%3CP%3Ebut%20when%20I%20run%20the%20query%20from%20monitor%20log%2C%20no%20data%20are%20returned%20even%20if%20there%20are%20records%20inside%20my%20actual%20log%20files.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Emy%20situation%20is%20%3A%3C%2FP%3E%3CP%3E1.%20I've%20upload%20the%20sample%20files%20on%20UTF-8%20encoded%20file%20and%20the%20actual%20log%20file%20are%20also%20UTF-8%20encoded%3C%2FP%3E%3CP%3E2.I%20put%20my%20log%20files%20on%20G%3A%5CLogs%5C%20folder%3C%2FP%3E%3CP%3E3.My%20VM%20are%20already%20connected%20to%20log%20analytics.%20Even%20tried%20to%20reconnect%20it%20once%20but%20nothing%20change.%3C%2FP%3E%3CP%3E4.Some%20people%20said%20that%20it%20will%20take%20a%20while%20for%20azure%20monitor%20to%20collect%20my%20custom%20log%20data.%20but%20it's%20been%202%20days%20and%20I%20don't%20think%20it%20would%20be%20taking%20that%20long.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eso%20I'm%20wondering%20if%20there%20are%20people%20in%20the%20community%20whose%20having%20the%20same%20problem%20as%20I%20do%2C%20or%20if%20there%20are%20someone%20who%20knows%20how%20to%20fix%20this.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eregards%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-631780%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Monitor%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ECustom%20Log%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EVirtual%20Machine%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EVirtual%20Machine%20Manager%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-632439%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-632439%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F346154%22%20target%3D%22_blank%22%3E%40orobmontana%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20can%20rule%20out%20latency%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%3C%2FA%3E%26nbsp%3Band%20you%20have%20refreshed%20your%20browser%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI'd%20try%20a%20craft%20from%20%3CU%3Escratch%3C%2FU%3E%20a%20simple%20log%20file%20in%20case%20some%20data%20in%20the%20example%20you%20are%20trying%20to%20send%20is%20causing%20an%20issue%20-%20control%20chars%20etc....%26nbsp%3B%201hr%20is%20the%20norm%2C%20maybe%20up%20to%208.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20criteria%20are%20really%20key%2C%20it%20sounds%20like%20you've%20checked%20already%2C%20but%20I've%20been%20caught%20out%20with%20the%20datetime%20format%20before...just%20this%20week%20I%20had%20a%20log%20with%20the%20right%20date%20format%20but%20the%20time%20was%20%22.%22%20separated.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20log%20files%20to%20be%20collected%20must%20match%20the%20following%20criteria.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CP%3EThe%20log%20must%20either%20have%20a%20single%20entry%20per%20line%20or%20use%20a%20timestamp%20matching%20one%20of%20the%20following%20formats%20at%20the%20start%20of%20each%20entry.%3C%2FP%3E%0A%3CP%3EYYYY-MM-DD%20HH%3AMM%3ASS%3CBR%20%2F%3EM%2FD%2FYYYY%20HH%3AMM%3ASS%20AM%2FPM%3CBR%20%2F%3EMon%20DD%2C%20YYYY%20HH%3AMM%3ASS%3CBR%20%2F%3EyyMMdd%20HH%3Amm%3Ass%3CBR%20%2F%3EddMMyy%20HH%3Amm%3Ass%3CBR%20%2F%3EMMM%20d%20hh%3Amm%3Ass%3CBR%20%2F%3Edd%2FMMM%2Fyyyy%3AHH%3Amm%3Ass%20zzz%3CBR%20%2F%3Eyyyy-MM-ddTHH%3Amm%3AssK%3C%2FP%3E%0A%3C%2FLI%3E%0A%3CLI%3E%3CP%3EThe%20log%20file%20must%20not%20allow%20circular%20logging%20or%20log%20rotation%2C%20where%20the%20file%20is%20overwritten%20with%20new%20entries.%3C%2FP%3E%0A%3C%2FLI%3E%0A%3CLI%3E%3CP%3EThe%20log%20file%20must%20use%20ASCII%20or%20UTF-8%20encoding.%20Other%20formats%20such%20as%20UTF-16%20are%20not%20supported.%3C%2FP%3E%0A%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-632450%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-632450%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%2C%20thank%20you%20for%20your%20input.%26nbsp%3B%3C%2FP%3E%3CP%3EI've%20checked%20my%20latency%20and%20it%20always%20between%201%2C75~3.%20so%20I%20don't%20think%20it%20would%20be%20a%20problem.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20actually%20creating%20a%20simple%20log%20file%20from%20scratch.%20it's%20contents%20are%3A%3C%2FP%3E%3CP%3E2019%2F05%2F08%2017%3A24%3A02.858%2CInterStage%2CWarning%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A02.879%2CPI%2CInfo%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A37.021%2CInterStage%2CWarning%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A37.023%2CPI%2CInfo%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A37.971%2CInterStage%2CWarning%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A37.974%2CPI%2CInfo%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A38.872%2CInterStage%2CWarning%2Ctest%20message%3CBR%20%2F%3E2019%2F05%2F08%2017%3A24%3A38.876%2CPI%2CInfo%2Ctest%20message%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eone%20thing%20that%20get%20in%20my%20mind%20is%2C%20even%20if%20I%20have%20a%20timestamp%20data%20on%20my%20record%2C%3C%2FP%3E%3CP%3Eif%20I%20chose%20single%20entry%20as%20my%20delimiter%20it's%20format%20shouldn't%20have%20effecting%20anything%20right%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eregards%2C%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-636261%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-636261%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI've%20tried%20and%20change%20my%20date%20format%20into%20YYYY-MM-DD%20just%20like%20you%20suggested.%3C%2FP%3E%3CP%3Ebut%20the%20result%20is%20same%2C%20log%20analytics%20workspace%20are%20showing%20my%20custom%20log%20name%20on%20the%20schema%2C%20but%20the%20data%20are%20not%20collected%20from%20my%20machine.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20don't%20think%20that%20the%20date%20format%20is%20the%20problem%20here%2C%20because%20when%20I%20tried%20to%20upload%20my%20sample%20files%2C%20the%20custom%20log%20wizard%20can%20read%20my%20files%20without%20problem.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-636693%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-636693%22%20slang%3D%22en-US%22%3EI%20just%20tried%20your%20file%2C%20both%20as%20a%20NEWLINE%20and%20a%20Date%20delimited%20custom%20log%20-%20with%20both%20date%20formats%2C%20and%20they%20both%20appeared%20in%20my%20workspace%20within%201hr%20(in%20West%20Europe)%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-654216%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-654216%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%20found%20the%20problem.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Emy%20VM%20network%20settings%20are%20blocking%20both%20inbound%20and%20outbound%20internet%20connection%2C%20which%20making%20the%20agent%20cannot%20collecting%20my%20custom%20log.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Efor%20people%20who%20having%20the%20same%20problem%20this%20work%20out%20for%20me%3A%3C%2FP%3E%3CP%3E1.%20set%20a%20proxy%20server%20for%20your%20VM%20to%20connect%20to%20the%20internet.%3C%2FP%3E%3CP%3E2.%20connect%20your%20VM%20to%20that%20proxy%20server%3C%2FP%3E%3CP%3E3.%20set%20up%20your%20agent%20to%20also%20connect%20using%20those%20proxy%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eand%20monitor%20will%20collect%20custom%20log%20normally.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3Bthanks%20a%20lot%20for%20your%20help%20mate%2C%20appreciate%20it!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eregards%2C%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-654573%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-654573%22%20slang%3D%22en-US%22%3ENo%20problem%2C%20and%20thanks%20for%20taking%20the%20time%20to%20provide%20a%20resolution%20and%20feedback.%20Cheers%20Clive%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1155048%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20not%20collecting%20custom%20log%20data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1155048%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHow%20can%20I%20change%20the%20time%20duration%20for%20which%20the%20custom%20log%20goes%20and%20reads%20logs%20from%20a%20txt%20file%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3BRegards%2C%3C%2FP%3E%3CP%3EMitesh%20Agrawal%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Occasional Contributor

Hi,

 

I'm creating a custom log on my windows VM and would like to query it on monitor log.

I've set up my custom log files just as Microsoft tell us to do on this archive

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs

 

all went well and my custom log are there in the log analytics schema,

but when I run the query from monitor log, no data are returned even if there are records inside my actual log files.

 

my situation is :

1. I've upload the sample files on UTF-8 encoded file and the actual log file are also UTF-8 encoded

2.I put my log files on G:\Logs\ folder

3.My VM are already connected to log analytics. Even tried to reconnect it once but nothing change.

4.Some people said that it will take a while for azure monitor to collect my custom log data. but it's been 2 days and I don't think it would be taking that long.

 

so I'm wondering if there are people in the community whose having the same problem as I do, or if there are someone who knows how to fix this.

 

regards,

 

6 Replies
Highlighted

@orobmontana 

 

If you can rule out latency https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time and you have refreshed your browser?

 

I'd try a craft from scratch a simple log file in case some data in the example you are trying to send is causing an issue - control chars etc....  1hr is the norm, maybe up to 8.  

 

The criteria are really key, it sounds like you've checked already, but I've been caught out with the datetime format before...just this week I had a log with the right date format but the time was "." separated.

 

The log files to be collected must match the following criteria.

  • The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry.

    YYYY-MM-DD HH:MM:SS
    M/D/YYYY HH:MM:SS AM/PM
    Mon DD, YYYY HH:MM:SS
    yyMMdd HH:mm:ss
    ddMMyy HH:mm:ss
    MMM d hh:mm:ss
    dd/MMM/yyyy:HH:mm:ss zzz
    yyyy-MM-ddTHH:mm:ssK

  • The log file must not allow circular logging or log rotation, where the file is overwritten with new entries.

  • The log file must use ASCII or UTF-8 encoding. Other formats such as UTF-16 are not supported.

Highlighted

@Clive Watson 

 

I've tried and change my date format into YYYY-MM-DD just like you suggested.

but the result is same, log analytics workspace are showing my custom log name on the schema, but the data are not collected from my machine.

 

I don't think that the date format is the problem here, because when I tried to upload my sample files, the custom log wizard can read my files without problem.

Highlighted
I just tried your file, both as a NEWLINE and a Date delimited custom log - with both date formats, and they both appeared in my workspace within 1hr (in West Europe)

Highlighted
Solution

Hi, found the problem.

 

my VM network settings are blocking both inbound and outbound internet connection, which making the agent cannot collecting my custom log.

 

for people who having the same problem this work out for me:

1. set a proxy server for your VM to connect to the internet.

2. connect your VM to that proxy server

3. set up your agent to also connect using those proxy

 

and monitor will collect custom log normally.

 

@Clive Watson thanks a lot for your help mate, appreciate it!

 

regards,

Highlighted
No problem, and thanks for taking the time to provide a resolution and feedback. Cheers Clive
Highlighted

Hi @Clive Watson,

 

How can I change the time duration for which the custom log goes and reads logs from a txt file?

 

 Regards,

Mitesh Agrawal