Negative Lookahead with Regular expression in Kusto Log Analytics

[SamiJ21]

Hello everyone,

I'm trying to extract exceptions within our logs using regular expression. And it happens that I need to perform a negative lookahead to ignore a specific string. The query looks like below:

let pattern = @'(?!Exception: System.Web.HttpUnhandledException)Exception: (.+)\s+Message: ([\S\s]+)\s+Source: ([\S\s]+)';
let standardize = (msg:string) {
    let msg2 = replace(@'"(\S+)"', '"xxx"', msg);
    replace(@"'(\S+)'", '"xxx"', msg2)
};
WebsiteLogs
| where ExceptionMessage contains "System.Web.HttpUnhandledException" or ExceptionMessage contains "[Extra Exception from config]"
| where  TimeGenerated > ago(7d) 
| project   URL,
            Exception=extract(pattern, 1, ExceptionMessage ), 
            Message = standardize(extract(pattern, 2, ExceptionMessage )),
            StackTrace = substring(extract(pattern, 3, ExceptionMessage ), 0, 500)
| summarize Errors_Count = count() by Message, URL, StackTrace
| sort by Errors_Count desc  nulls last 
| limit 1000

A sample of exception below is like below:

Exception: System.Web.HttpUnhandledException
Message: An unhandled exception occurred.
Source: Sitecore.Mvc
at Sitecore.Mvc.Pipelines.MvcEvents.Exception.ShowAspNetErrorMessage.ShowErrorMessage(ExceptionContext exceptionContext, ExceptionArgs args)
at Sitecore.Mvc.Pipelines.MvcEvents.Exception.ShowAspNetErrorMessage.Process(ExceptionArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
at Sitecore.Mvc.Pipelines.PipelineService.RunPipeline[TArgs](String pipelineName, TArgs args)
at Sitecore.Mvc.Filters.PipelineBasedRequestFilter.OnException(ExceptionContext exceptionContext)
at System.Web.Mvc.ControllerActionInvoker.InvokeExceptionFilters(ControllerContext controllerContext, IList`1 filters, Exception exception)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
at Sitecore.Mvc.Controllers.ControllerRunner.ExecuteController(Controller controller)
at Sitecore.Mvc.Controllers.ControllerRunner.Execute()
at Sitecore.Mvc.Presentation.ControllerRenderer.Render(TextWriter writer)
at Sitecore.Mvc.Pipelines.Response.RenderRendering.ExecuteRenderer.Render(Renderer renderer, TextWriter writer, RenderRenderingArgs args)

Nested Exception

Exception: System.ArgumentNullException
Message: Value cannot be null.
Parameter name: item
Source: Website.Module.Product

As you can see, I want to ignore the System.Web.HttpUnhandledException as it Is not very informative for my case. The information that I want to extract is found below the Nested Exception. As such, I'm using regex negative lookahead for this, and it match very well when tested on a regex tester. But the thing is it seems that Log Analytics doesn't support this feature. Is it the case or I missed something in the Kusto syntax for regular expressions?

Thank you in advance for your precious help 

[#2]

Well, I managed to get over it without using regex negative lookhead by using another custom function to skip the non-relevant information. My solution looks like below in case someone else is facing similar case

let pattern = @'Exception: (.+)\s+Message: ([\S\s]+)\s+Source: ([\S\s]+)';
let standardize = (msg:string) {
    let msg2 = replace(@'"(\S+)"', '"xxx"', msg);
    replace(@"'(\S+)'", '"xxx"', msg2)
};
let getException = (msg: string) {
    case(msg contains "System.Web.HttpUnhandledException", substring(msg, indexof(msg, "Nested Exception") + 17), msg)
};
WebsiteLogs
| where ExceptionMessage contains "System.Web.HttpUnhandledException" or ExceptionMessage contains "[Extra Exception from config]"
| where  TimeGenerated > ago(7d) 
| project   URL,
            Exception=extract(pattern, 1, ExceptionMessage ), 
            Message = standardize(extract(pattern, 2, getException(ExceptionMessage) )),
            StackTrace = substring(extract(pattern, 3, getException(ExceptionMessage )), 0, 500)
| summarize Errors_Count = count() by Message, URL, StackTrace
| sort by Errors_Count desc  nulls last 
| limit 1000

Here, the getException() function just skip the ExceptionMessage till the first Nested Exception for logs containing the System.Web.HttpUnhandledException.

Nevertheless, I'm still wondering if regex lookahead is supported or not

[#3]

Thanks for sharing Sami. Meanwhile I checked with the Kusto group and that's indeed not supported.

[#4]

Apparently the full explanation is here. Quoting from it:

"Kusto uses the re2 library: https://github.com/google/re2/wiki/Syntax, as mentioned here: https://docs.microsoft.com/en-us/azure/kusto/query/re2

What you're trying to achieve isn't supported by that library - see https://github.com/google/re2/wiki/WhyRE2:

'As a matter of principle, RE2 does not support constructs for which only backtracking solutions are known to exist. Thus, backreferences and look-around assertions are not supported.'"

[#5]

@Noa Kuperberg thanks for the information on the RE2 library. Negative lookahead would be very useful in some cases. :\ would be great if this could be added somehow. Why not use System.Text.RegularExpressions instead of re2? :p