Azure not collecting custom log data

[orobmontana]

Hi,

I'm creating a custom log on my windows VM and would like to query it on monitor log.

I've set up my custom log files just as Microsoft tell us to do on this archive

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs

all went well and my custom log are there in the log analytics schema,

but when I run the query from monitor log, no data are returned even if there are records inside my actual log files.

my situation is :

1. I've upload the sample files on UTF-8 encoded file and the actual log file are also UTF-8 encoded

2.I put my log files on G:\Logs\ folder

3.My VM are already connected to log analytics. Even tried to reconnect it once but nothing change.

4.Some people said that it will take a while for azure monitor to collect my custom log data. but it's been 2 days and I don't think it would be taking that long.

so I'm wondering if there are people in the community whose having the same problem as I do, or if there are someone who knows how to fix this.

regards,

[#2]

@orobmontana 

If you can rule out latency https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time and you have refreshed your browser?

I'd try a craft from scratch a simple log file in case some data in the example you are trying to send is causing an issue - control chars etc....  1hr is the norm, maybe up to 8.  

The criteria are really key, it sounds like you've checked already, but I've been caught out with the datetime format before...just this week I had a log with the right date format but the time was "." separated.

The log files to be collected must match the following criteria.

• The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry.

  YYYY-MM-DD HH:MM:SS
  M/D/YYYY HH:MM:SS AM/PM
  Mon DD, YYYY HH:MM:SS
  yyMMdd HH:mm:ss
  ddMMyy HH:mm:ss
  MMM d hh:mm:ss
  dd/MMM/yyyy:HH:mm:ss zzz
  yyyy-MM-ddTHH:mm:ssK

• The log file must not allow circular logging or log rotation, where the file is overwritten with new entries.

• The log file must use ASCII or UTF-8 encoding. Other formats such as UTF-16 are not supported.

[#3]

@CliveWatson 

I've tried and change my date format into YYYY-MM-DD just like you suggested.

but the result is same, log analytics workspace are showing my custom log name on the schema, but the data are not collected from my machine.

I don't think that the date format is the problem here, because when I tried to upload my sample files, the custom log wizard can read my files without problem.

[#4]

I just tried your file, both as a NEWLINE and a Date delimited custom log - with both date formats, and they both appeared in my workspace within 1hr (in West Europe)

[#5 · Accepted solution]

Hi, found the problem.

my VM network settings are blocking both inbound and outbound internet connection, which making the agent cannot collecting my custom log.

for people who having the same problem this work out for me:

1. set a proxy server for your VM to connect to the internet.

2. connect your VM to that proxy server

3. set up your agent to also connect using those proxy

and monitor will collect custom log normally.

@CliveWatson thanks a lot for your help mate, appreciate it!

regards,

[#6]

No problem, and thanks for taking the time to provide a resolution and feedback. Cheers Clive

[#7]

Hi @CliveWatson,

How can I change the time duration for which the custom log goes and reads logs from a txt file?

 Regards,

Mitesh Agrawal

[#8]

I changed the encoding of the log txt file to UTF-8 and it started capturing the logs in Azure monitor via Log Analytics workspace (via monitoring agent). 

These are the steps followed.

1) From the Log Analytic Workspace > Virtual Machines > Activated the workspace by 'Connect' option. This will install Microsoft Monitoring agent in the VM and register with this Log analytics workspace Id. ( see this by Login to VM and Control panel > Microsoft Monitoring agent > Azure Log Analytics (OMS) )

2) Then from Log analytics workspace > Custom Logs > Add log > follow the steps.

No need of (inbound outbound 443 https rules

No need of dependency agent to be enabled.

Within 10 to 15 minutes logs start the capturing.

Log file creation,

$logfilepath=$MyInvocation.MyCommand.Path -replace '\.ps1$', '.txt'
function WriteToLogFile ($message)
{
  (Get-Date).ToString() + " : " + $message | Out-File $logfilepath -Append -Encoding utf8
}

[#9]

Hi All,

We have been useing Log Analytics to track all events/Opernatins of SSAS every thing is working fine but since few days the below operations.

But some how some of the Operations are not logging like 'Syncdatabase' and 'memory_metric'

name_s == ""memory_metric""

OperationName == ""SyncDatabase""

Need some help someone who knows how to fix this.

I did not find much forum discussions on Log Analytics.

From all above discussions ,i got that nothing looging for your case.But we are missing only few events/Opernatins.

Thanks.