OMSAgent - CEF logs are sent but not appearing in Sentinel

[csmits]

Hi,

We are trying to forward CEF logs to Sentinel using an oms-agent instance. We have successfully onboarded the logs at first, but after about an hour, logs stopped appearing.

We have turned on the debug logs for the agent, which showed that logs were being sent successfully to the workspace. Furthermore, heartbeat logs keep appearing in Sentinel too.

Rsyslog is properly configured, and tcpdump indeed also shows traffic as expected.

Any idea what may cause the logs to stop appearing in the log analytics workspace?

Thanks in advance!

[#2]

@csmits I suspect this might have to do proper parsing. How are your forwarding rules configured on the originating device? What type of device is it? Had something similar happen working with a customer recently which led to this blog post:

https://secureinfra.blog/2020/07/06/tips-for-parsing-syslog-to-azure-sentinel/

[#3]

I was having transient issue yesterday where I would load CEF logs and not be able to query them until 1hr later even though the ingestion_time() indicated they only took 20s to arrive. Shortly after all the logs disappeared again... it was another hour after that before I could see them all again :S.

This was the first time loading CEF logs into this account, perhaps there's some latency in the setup process for the first logs that come through 🤷‍:male_sign:

[#4]

@Rod_TrentThanks for the insight. It is a Check Point device, and the "Check Point" connector has turned green and is thus active. I suspect the parsing is okay, because ingestion does happen.

However, it looks like the ingestion is hitting some rate limits. Logs start reappearing every day between 12:00 and 13:00, after which they stop showing for 24 hours. This is a repetitive cycle. I will check back to see what kind of response is sent when data is ingested (the omsagent logs still show: "successfully sent logs").

[#5]

@csmits : I think such an issue is hard to resolve in the community and is very important for us to resolve. Can you open a support ticket?