How to solve the problem of displaying multiline logs from the AKS cluster in Log Analytics?

[prokhorovd]

I have two problems - the multiline logs of my AKS containers comes to the LogsAnalytics table as separate entries. And these records are also recorded out of order.

The original logs is:

2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92.......
System.Exception: Incorrect groupId or reportId for report: [....
at EffieX.Domain.Services.Dashboard.DashboardSer...
at EffieX.WebAPI.Controllers.Dashboa....
at Microsoft.AspNetCore.Mvc.Internal.Action...
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerA..
at Microsoft.AspNetCore.Mvc.Internal.ControllerActi...
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)..

In Logs Analytics I see:

1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ActionMetho...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Rout....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Inte....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.N...
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.SwaggerUI.Swa...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker....
1/29/2020, 2:46:12.928 PM at EffieX.Domain.Services.Dashboard.DashboardService.GetAsync(String ..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ControllerA.....
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.Swagger.SwaggerMiddl....
1/29/2020, 2:46:12.928 PM 2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92188f3_a8af656f-63e..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Routing.EndpointMiddlew..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Authentication.AuthenticationMiddle..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Con..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Builder.Extensions.MapWh..
1/29/2020, 2:46:12.928 PM at System.Threading.Tasks.ValueTask`1.get_Result()
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Controlle..
1/29/2020, 2:46:12.928 PM System.Exception: Incorrect groupId or reportId for report: [ x..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
1/29/2020, 2:46:12.928 PM at .re.Invoke(HttpContext context)
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Cont.
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Resourc.

Is there any solution?

Thanks

[#2]

Hi @prokhorovd, that's really strange.

I'm checking that with the AKS team, will get back to you as soon as I get their reply.

Noa

[#3]

@prokhorovd , Container runtime picks up the log as is, and we (AKS Monitoring) pick up as-is from container runtime. In this case, each line in the stack is a seperate line (with a \n). Workaround people do is to log as a json. https://docs.microsoft.com/en-us/azure/azure-monitor/faq#how-do-i-enable-multi-line-logging

We will look into stitching intelligently up-stream at some point in the near future.

[#4]

Hi, we’re excited to let you know that Azure Monitor – Container Insights now supports multiline logging! With this feature enabled, previously split container logs are stitched together and sent as single entries to the ContainerLogV2 table. For more information and instructions on how to enable, please visit our documentation here: https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-logging-v2?tabs=....