How to solve the problem of displaying multiline logs from the AKS cluster in Log Analytics?

Senior Member

I have two problems - the multiline logs of my AKS containers comes to the LogsAnalytics table as separate entries. And these records are also recorded out of order.

The original logs is:

 

2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92.......
System.Exception: Incorrect groupId or reportId for report: [....
at EffieX.Domain.Services.Dashboard.DashboardSer...
at EffieX.WebAPI.Controllers.Dashboa....
at Microsoft.AspNetCore.Mvc.Internal.Action...
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerA..
at Microsoft.AspNetCore.Mvc.Internal.ControllerActi...
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)..

 

In Logs Analytics I see:

1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ActionMetho...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Rout....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Inte....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.N...
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.SwaggerUI.Swa...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker....
1/29/2020, 2:46:12.928 PM at EffieX.Domain.Services.Dashboard.DashboardService.GetAsync(String ..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ControllerA.....
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.Swagger.SwaggerMiddl....
1/29/2020, 2:46:12.928 PM 2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92188f3_a8af656f-63e..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Routing.EndpointMiddlew..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Authentication.AuthenticationMiddle..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Con..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Builder.Extensions.MapWh..
1/29/2020, 2:46:12.928 PM at System.Threading.Tasks.ValueTask`1.get_Result()
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Controlle..
1/29/2020, 2:46:12.928 PM System.Exception: Incorrect groupId or reportId for report: [ x..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
1/29/2020, 2:46:12.928 PM at .re.Invoke(HttpContext context)
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Cont.
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Resourc.

 

 

Is there any solution?

Thanks

2 Replies

Hi @prokhorovd, that's really strange.

I'm checking that with the AKS team, will get back to you as soon as I get their reply.

 

Noa

@prokhorovd , Container runtime picks up the log as is, and we (AKS Monitoring) pick up as-is from container runtime. In this case, each line in the stack is a seperate line (with a \n). Workaround people do is to log as a json. https://docs.microsoft.com/en-us/azure/azure-monitor/faq#how-do-i-enable-multi-line-logging

 

We will look into stitching intelligently up-stream at some point in the near future.