Azure sentinel log data fields missing

[TarunV12]

HI Team,

WE have observed that some of the fields are missing after we normalize the data in Azure Sentinel. Where customer want some other fields also need to show in Normalized data.

If we ingest payload data into the log Analytics the cost is going double.

Could we have  solution for this.

Regards,

Tv19

[#2]

Hi, which data sources and fields? Which Tables are you using i.e CommonSecurityLog and the missing data is 'a column called, "my data"'. A screenshot and samples would be helpful? How have you normalized the data, is this your own function or one or one (or all) of the provided ones?

Thanks

[#3]

Hey any updates on this?

My use case is similar. I'm using the standard CEF log format coming from the PaloAlto to the Azure Log Agent server. It is successfully sending logs into my Azure Log Analytics Workspace and connected successfully to Azure Sentinel.

The issue I am having is that when I search the CommonSecurityLog table, I don't see certain columns like "source zone" or "destination zone" and many others, but I do see columns like source and target IP..

[#4]

@AndrewX this should come through as a device custom field for Palo Alto, if not does the field appear in the `AdditionalExtensions` field?

Field	Value
DeviceCustomString4	UNTRUSTED
DeviceCustomString4Label	Source Zone
DeviceCustomString5	TRUSTED
DeviceCustomString5Label	Destination Zone

E.g. for some of our logs we want the `ThreatCategory` but it's not a top level field so we need to extract it from `AdditionalExtensions`.

CommonSecurityLog
| extend ThreatCategory = extract(@"PanOSThreatCategory=(.*?);", 1, AdditionalExtensions)

[#5]

@pemontto hello and thank you for your reply.  I do have those columns it appears, but they seem to be blank?

This is the output of a syslog message come off the PA, being received at the Azure log agent server using the following command "tcpdump port 514 -v"

cs4Label=Source Zone cs4=UNTRUSTED cs5Label=Destination Zone cs5=TRUSTED

 Is my custom format wrong? Should i amend the formatting output to be the full "DeviceCustomString4" instead of "cs4"

[#6]

@pemontto this is my CEF format

CEF:0|Palo Alto Networks|PAN-OS|7.0.0|$subtype|$type|1|rt=$cefformatted-receive_time deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser duser=$dstuser app=$app cs4Label=Source Zone cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport flexString1Label=Flags flexString1=$flags proto=$proto act=$action flexNumber1Label=Total bytes flexNumber1=$bytes in=$bytes_sent out=$bytes_received cn2Label=Packets cn2=$packets PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent start=$cef-formatted-time_generated cs2Label=URL Category cs2=$category externalId=$seqno reason=$session_end_reason PanOSVsysName=$vsys_name dvchost=$device_name cat=$action_source

[#7]

Sorry can’t tell you offhand. However the Palo CEF docs are terrible. They only provide the PDF and we continually see customers copy paste verbatim with the incorrect formatting and end up with useless logs!

See how it compares to these cleaned up versions here https://github.com/pemontto/Palo-Alto-CEF

If that checks out then it would seem the Sentinel CEF parser doesn’t like your logs. Would suggest opening a ticket with MS then.