How to solve the problem of displaying multiline logs from the AKS cluster in Log Analytics?

%3CLINGO-SUB%20id%3D%22lingo-sub-1139433%22%20slang%3D%22en-US%22%3EHow%20to%20solve%20the%20problem%20of%20displaying%20multiline%20logs%20from%20the%20AKS%20cluster%20in%20Log%20Analytics%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1139433%22%20slang%3D%22en-US%22%3E%3CP%3EI%20have%20two%20problems%20-%20the%20multiline%20logs%20of%20my%20AKS%20containers%20comes%20to%20the%20LogsAnalytics%20table%20as%20separate%20entries.%20And%20these%20records%20are%20also%20recorded%20out%20of%20order.%3C%2FP%3E%3CP%3EThe%20original%20logs%20is%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E2020-01-29%2012%3A46%3A12%20%5BERR%5D%20Trace%20ID%3A%20%5Bbd54c0a3-ace3-496c-9a5b-12e5e92.......%3CBR%20%2F%3ESystem.Exception%3A%20Incorrect%20groupId%20or%20reportId%20for%20report%3A%20%5B....%3CBR%20%2F%3Eat%20EffieX.Domain.Services.Dashboard.DashboardSer...%3CBR%20%2F%3Eat%20EffieX.WebAPI.Controllers.Dashboa....%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.Action...%3CBR%20%2F%3Eat%20System.Threading.Tasks.ValueTask%601.get_Result()%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ControllerA..%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ControllerActi...%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()%3CBR%20%2F%3Eat%20Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext%20context)..%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20Logs%20Analytics%20I%20see%3A%3C%2FP%3E%3CP%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.ActionMetho...%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Rout....%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Inte....%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.N...%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Swashbuckle.AspNetCore.SwaggerUI.Swa...%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker....%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20EffieX.Domain.Services.Dashboard.DashboardService.GetAsync(String%20..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.ControllerA.....%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Swashbuckle.AspNetCore.Swagger.SwaggerMiddl....%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%202020-01-29%2012%3A46%3A12%20%5BERR%5D%20Trace%20ID%3A%20%5Bbd54c0a3-ace3-496c-9a5b-12e5e92188f3_a8af656f-63e..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Routing.EndpointMiddlew..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Authentication.AuthenticationMiddle..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.Con..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Builder.Extensions.MapWh..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20System.Threading.Tasks.ValueTask%601.get_Result()%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.Controlle..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20System.Exception%3A%20Incorrect%20groupId%20or%20reportId%20for%20report%3A%20%5B%20x..%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20.re.Invoke(HttpContext%20context)%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.Cont.%3CBR%20%2F%3E1%2F29%2F2020%2C%202%3A46%3A12.928%20PM%20at%20Microsoft.AspNetCore.Mvc.Internal.Resourc.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EIs%20there%20any%20solution%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EThanks%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1139433%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1210815%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20solve%20the%20problem%20of%20displaying%20multiline%20logs%20from%20the%20AKS%20cluster%20in%20Log%20Analytics%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1210815%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F538519%22%20target%3D%22_blank%22%3E%40prokhorovd%3C%2FA%3E%2C%20that's%20really%20strange.%3C%2FP%3E%0A%3CP%3EI'm%20checking%20that%20with%20the%20AKS%20team%2C%20will%20get%20back%20to%20you%20as%20soon%20as%20I%20get%20their%20reply.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENoa%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1211803%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20solve%20the%20problem%20of%20displaying%20multiline%20logs%20from%20the%20AKS%20cluster%20in%20Log%20Analytics%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1211803%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F538519%22%20target%3D%22_blank%22%3E%40prokhorovd%3C%2FA%3E%26nbsp%3B%2C%20Container%20runtime%20picks%20up%20the%20log%20as%20is%2C%20and%20we%20(AKS%20Monitoring)%20pick%20up%20as-is%20from%20container%20runtime.%20In%20this%20case%2C%20each%20line%20in%20the%20stack%20is%20a%20seperate%20line%20(with%20a%20%5Cn).%20Workaround%20people%20do%20is%20to%20log%20as%20a%20json.%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Ffaq%23how-do-i-enable-multi-line-logging%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Ffaq%23how-do-i-enable-multi-line-logging%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20will%20look%20into%20stitching%20intelligently%20up-stream%20at%20some%20point%20in%20the%20near%20future.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Senior Member

I have two problems - the multiline logs of my AKS containers comes to the LogsAnalytics table as separate entries. And these records are also recorded out of order.

The original logs is:

 

2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92.......
System.Exception: Incorrect groupId or reportId for report: [....
at EffieX.Domain.Services.Dashboard.DashboardSer...
at EffieX.WebAPI.Controllers.Dashboa....
at Microsoft.AspNetCore.Mvc.Internal.Action...
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerA..
at Microsoft.AspNetCore.Mvc.Internal.ControllerActi...
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)..

 

In Logs Analytics I see:

1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ActionMetho...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Rout....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Inte....
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.N...
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.SwaggerUI.Swa...
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker....
1/29/2020, 2:46:12.928 PM at EffieX.Domain.Services.Dashboard.DashboardService.GetAsync(String ..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ControllerA.....
1/29/2020, 2:46:12.928 PM at Swashbuckle.AspNetCore.Swagger.SwaggerMiddl....
1/29/2020, 2:46:12.928 PM 2020-01-29 12:46:12 [ERR] Trace ID: [bd54c0a3-ace3-496c-9a5b-12e5e92188f3_a8af656f-63e..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Routing.EndpointMiddlew..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Authentication.AuthenticationMiddle..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Con..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Builder.Extensions.MapWh..
1/29/2020, 2:46:12.928 PM at System.Threading.Tasks.ValueTask`1.get_Result()
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Controlle..
1/29/2020, 2:46:12.928 PM System.Exception: Incorrect groupId or reportId for report: [ x..
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
1/29/2020, 2:46:12.928 PM at .re.Invoke(HttpContext context)
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Cont.
1/29/2020, 2:46:12.928 PM at Microsoft.AspNetCore.Mvc.Internal.Resourc.

 

 

Is there any solution?

Thanks

2 Replies

Hi @prokhorovd, that's really strange.

I'm checking that with the AKS team, will get back to you as soon as I get their reply.

 

Noa

@prokhorovd , Container runtime picks up the log as is, and we (AKS Monitoring) pick up as-is from container runtime. In this case, each line in the stack is a seperate line (with a \n). Workaround people do is to log as a json. https://docs.microsoft.com/en-us/azure/azure-monitor/faq#how-do-i-enable-multi-line-logging

 

We will look into stitching intelligently up-stream at some point in the near future.