CEF ingestion not parsing all fields or field limitations

[Jordan Mills]

I'm trying to get data from palo alto firewalls into sentinel. After fighting through some major issues with documentation and incorrect automatic configuration, we have the data getting into the CommonSecurityLog table.  The problem is that most of the fields needed for the solution components are not parsed.  They're just stuck in in the AdditionalExtensions column.

The end result is that the resources like playbooks included with the connector are useless.  They expect and rely on fields like ReceiptTime, LogSeverity, DeviceAction, DeviceCustomString2, DestinationPort, DestinationIP, Message, SourcePort, SourceIP, DestinationUserID, RequestURL.  But none of them are parsed out.  These all match appropriate CommonSecurityLog fields or the CEF name matching table at https://learn.microsoft.com/en-us/azure/sentinel/cef-name-mapping

In https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/PaloAlto-PAN-OS/Data%20Connectors/read... referred to in the connector page, I noted that implies the agent can only parse seven CEF fields, but it's in in broken english so it's not clear if this is a continuation of the poor documentation or an actual limitation (that I haven't seen referred to anywhere else).  

Is the seven fields limitation present in the agent?  If it is, why were resources like workbooks built out to depend on more fields?  If it isn't a limitation, how do I get the other fields parsed?

[#2]

Have you checked out the ASIM parser functions that were created to extract this information as needed to see if they provide the information you need?

[#3]

Yes. They expect these fields to be there. That can easily be verified by looking at the function code. The important fields, like MaliciousIP, ThreatSeverity, and ReportReferenceLink are empty (for the output of _Im_NetworkSession_PaloAltoCEFV05, in this case).

And nothing seems to use these functions any way. The workbooks I've saved from templates all have their own direct queries of the CommonSecurityLog table. Of course they all have similar issues.

Even if the ASIM functions worked and were used elsewhere, their basis of performing string extraction at query time makes no sense. I get that Kusto performs indexing in ways that might not be expected by someone used to SQL Server, but I can't imagine that query-time parsing and partial indexing of string fields can ever be more efficient than a simple string parsing at ingestion.