Forum Discussion

SamiJ21's avatar
SamiJ21
Copper Contributor
Feb 20, 2020

Negative Lookahead with Regular expression in Kusto Log Analytics

Hello everyone,

 

I'm trying to extract exceptions within our logs using regular expression. And it happens that I need to perform a negative lookahead to ignore a specific string. The query looks like below:

 

 

 

 

let pattern = @'(?!Exception: System.Web.HttpUnhandledException)Exception: (.+)\s+Message: ([\S\s]+)\s+Source: ([\S\s]+)';
let standardize = (msg:string) {
    let msg2 = replace(@'"(\S+)"', '"xxx"', msg);
    replace(@"'(\S+)'", '"xxx"', msg2)
};
WebsiteLogs
| where ExceptionMessage contains "System.Web.HttpUnhandledException" or ExceptionMessage contains "[Extra Exception from config]"
| where  TimeGenerated > ago(7d) 
| project   URL,
            Exception=extract(pattern, 1, ExceptionMessage ), 
            Message = standardize(extract(pattern, 2, ExceptionMessage )),
            StackTrace = substring(extract(pattern, 3, ExceptionMessage ), 0, 500)
| summarize Errors_Count = count() by Message, URL, StackTrace
| sort by Errors_Count desc  nulls last 
| limit 1000

 

 

 
A sample of exception below is like below:

 

 

Exception: System.Web.HttpUnhandledException
Message: An unhandled exception occurred.
Source: Sitecore.Mvc
at Sitecore.Mvc.Pipelines.MvcEvents.Exception.ShowAspNetErrorMessage.ShowErrorMessage(ExceptionContext exceptionContext, ExceptionArgs args)
at Sitecore.Mvc.Pipelines.MvcEvents.Exception.ShowAspNetErrorMessage.Process(ExceptionArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
at Sitecore.Mvc.Pipelines.PipelineService.RunPipeline[TArgs](String pipelineName, TArgs args)
at Sitecore.Mvc.Filters.PipelineBasedRequestFilter.OnException(ExceptionContext exceptionContext)
at System.Web.Mvc.ControllerActionInvoker.InvokeExceptionFilters(ControllerContext controllerContext, IList`1 filters, Exception exception)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
at Sitecore.Mvc.Controllers.ControllerRunner.ExecuteController(Controller controller)
at Sitecore.Mvc.Controllers.ControllerRunner.Execute()
at Sitecore.Mvc.Presentation.ControllerRenderer.Render(TextWriter writer)
at Sitecore.Mvc.Pipelines.Response.RenderRendering.ExecuteRenderer.Render(Renderer renderer, TextWriter writer, RenderRenderingArgs args)

Nested Exception

Exception: System.ArgumentNullException
Message: Value cannot be null.
Parameter name: item
Source: Website.Module.Product

 

 

As you can see, I want to ignore the System.Web.HttpUnhandledException as it Is not very informative for my case. The information that I want to extract is found below the Nested Exception. As such, I'm using regex negative lookahead for this, and it match very well when tested on a regex tester. But the thing is it seems that Log Analytics doesn't support this feature. Is it the case or I missed something in the Kusto syntax for regular expressions?
 
Thank you in advance for your precious help :smile:
azure monitor
Query Language

4 Replies

  • Noa Kuperberg's avatar
    Noa Kuperberg
    Icon for Microsoft rankMicrosoft
    Feb 25, 2020

    Apparently the full explanation is here. Quoting from it:

     

    "Kusto uses the re2 library: https://github.com/google/re2/wiki/Syntax, as mentioned here: https://docs.microsoft.com/en-us/azure/kusto/query/re2

    What you're trying to achieve isn't supported by that library - see https://github.com/google/re2/wiki/WhyRE2:

    'As a matter of principle, RE2 does not support constructs for which only backtracking solutions are known to exist. Thus, backreferences and look-around assertions are not supported.'"

    • frmax's avatar
      frmax
      Copper Contributor
      Jan 26, 2021

      Noa Kuperberg thanks for the information on the RE2 library. Negative lookahead would be very useful in some cases. 😕 would be great if this could be added somehow. Why not use System.Text.RegularExpressions instead of re2? 😛

       

  • Noa Kuperberg's avatar
    Noa Kuperberg
    Icon for Microsoft rankMicrosoft
    Feb 24, 2020
    Thanks for sharing Sami. Meanwhile I checked with the Kusto group and that's indeed not supported.
  • SamiJ21's avatar
    SamiJ21
    Copper Contributor
    Feb 20, 2020

    Well, I managed to get over it without using regex negative lookhead by using another custom function to skip the non-relevant information. My solution looks like below in case someone else is facing similar case :smile:

    let pattern = @'Exception: (.+)\s+Message: ([\S\s]+)\s+Source: ([\S\s]+)';
let standardize = (msg:string) {
    let msg2 = replace(@'"(\S+)"', '"xxx"', msg);
    replace(@"'(\S+)'", '"xxx"', msg2)
};
let getException = (msg: string) {
    case(msg contains "System.Web.HttpUnhandledException", substring(msg, indexof(msg, "Nested Exception") + 17), msg)
};
WebsiteLogs
| where ExceptionMessage contains "System.Web.HttpUnhandledException" or ExceptionMessage contains "[Extra Exception from config]"
| where  TimeGenerated > ago(7d) 
| project   URL,
            Exception=extract(pattern, 1, ExceptionMessage ), 
            Message = standardize(extract(pattern, 2, getException(ExceptionMessage) )),
            StackTrace = substring(extract(pattern, 3, getException(ExceptionMessage )), 0, 500)
| summarize Errors_Count = count() by Message, URL, StackTrace
| sort by Errors_Count desc  nulls last 
| limit 1000

    Here, the getException() function just skip the ExceptionMessage till the first Nested Exception for logs containing the System.Web.HttpUnhandledException.

     

    Nevertheless, I'm still wondering if regex lookahead is supported or not :smile:

     

     

Resources