OMS Agent on Azure Sentinel Log forwarder not receiving and forwarding logs to sentinel workspace

%3CLINGO-SUB%20id%3D%22lingo-sub-2851297%22%20slang%3D%22en-US%22%3EOMS%20Agent%20on%20Azure%20Sentinel%20Log%20forwarder%20not%20receiving%20and%20forwarding%20logs%20to%20sentinel%20workspace%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2851297%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20have%20observed%20that%20we%20no%20longer%20are%20receiving%20Syslog%20and%20CEF%20logs%20from%20the%20Azure%20Sentinel%20Log%20forwarder%20that%20is%20deployed%20on%20client%20premise.%20I%20have%20performed%20the%20following%20steps%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Enetstat%20-an%20%7C%20grep%20514%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EStatus%3A%20Listening%20or%20established%20(which%20is%20fine)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Enetstat%20-an%20%7C%20grep%2025226%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EStatus%3A%20Listening%20or%20established%20(which%20is%20fine)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Esudo%20tcpdump%20-A%20-ni%20any%20port%20514%20-vv%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EStatus%3A%20receive%20logs%20from%20the%20data%20sources%20(which%20is%20fine)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Esudo%20tcpdump%20-A%20-ni%20any%20port%20514%20-vv%20%7C%20grep%20(Zscaler%20IP)%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EStatus%3A%20receive%20logs%20from%20the%20Zscaler%20data%20source%2C%20the%20logs%20showed%20Palo%20Alto%20name%20in%20the%20CEF%20messages%20which%20means%20Zscaler%20traffic%20was%20routed%20through%20the%20firewall%20(which%20is%20fine%2C%20as%20confirmed%20by%20client)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Esudo%20tcpdump%20-A%20-ni%20any%20port%2025226%20-vv%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3E%3CFONT%20color%3D%22%23FF0000%22%3EStatus%3A%20No%20logs%20were%20received%20(Issue%20Identified)%3C%2FFONT%3E%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3Esudo%20tcpdump%20-A%20-ni%20any%20port%2025226%20-vv%20%7C%20grep%20(Zscaler%20IP)%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CFONT%20color%3D%22%23FF0000%22%3E%3CSTRONG%3EStatus%3A%20No%20logs%20were%20received%20(Issue%20Identified)%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3ERestarted%20the%20Rsyslog%20Service%3A%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3Eservice%20rsyslog%20restart%20(After%20service%20restart%2C%20Azure%20Sentinel%20Started%20receiving%20the%20syslog.%20The%20Syslog%20data%20source%20came%20up%20and%20working%20fine)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3ERestarted%20the%20OMSAgent%20Service%3C%2FP%3E%3CP%20class%3D%22%22%3E%2Fopt%2Fmicrosoft%2Fomsagent%2Fbin%2Fservice_control%20restart%20%7Bworkspace%20ID%7D%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EStatus%3A%20There%20was%20no%20status%20message%20and%20prompt%20came%2C%20assuming%20it%20restarted%20in%20the%20background%20(%3CFONT%20color%3D%22%23FF0000%22%3EPlease%20confirm%20if%20this%20is%20the%20normal%2C%20not%20prompting%20or%20showing%20any%20message%3C%2FFONT%3E)%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3E%3CSTRONG%3EAfter%20OMS%20Agent%20restart%2C%20ran%20tcpdump%20again%20on%20OMS%20Agent%20to%20see%20if%20it%20starts%20receiving%20the%20logs%20but%20no%20luck.%3C%2FSTRONG%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3EI%20followed%20the%20following%20link%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Ftroubleshooting-cef-syslog%3Ftabs%3Drsyslog%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Ftroubleshooting-cef-syslog%3Ftabs%3Drsyslog%3C%2FA%3E%3C%2FP%3E%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22%22%3ECan%20any%20one%20guide%20what%20probably%20be%20the%20cause%20of%20this%20issue%3F%3F%3F%3F%3F%20Any%20help%20will%20be%20much%20appreciated.%20Thanks%20in%20advance.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2856542%22%20slang%3D%22en-US%22%3ERe%3A%20OMS%20Agent%20on%20Azure%20Sentinel%20Log%20forwarder%20not%20receiving%20and%20forwarding%20logs%20to%20sentinel%20workspac%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2856542%22%20slang%3D%22en-US%22%3EI%20would%20also%20check%20with%20your%20Azure%20admins%20to%20see%20if%20they%20may%20have%20added%20any%20firewall%20rules%20that%20could%20have%20blocked%20the%20data.%20It%20seems%20that%20there%20is%20one%20command%20from%20the%20document%20to%20check%20for%20this%3A%3CBR%20%2F%3Esudo%20wget%20-O%20cef_troubleshoot.py%20%3CA%20href%3D%22https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_troubleshoot.py%26amp%3B%26amp%3Bsudo%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_troubleshoot.py%26amp%3B%26amp%3Bsudo%3C%2FA%3E%20python%20cef_troubleshoot.py%20%5BWorkspaceID%5D%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2869118%22%20slang%3D%22en-US%22%3ERe%3A%20OMS%20Agent%20on%20Azure%20Sentinel%20Log%20forwarder%20not%20receiving%20and%20forwarding%20logs%20to%20sentinel%20workspac%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2869118%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3B%20Thank%20you%20for%20the%20prompt%20response.%20We%20have%20got%20the%20issue%20resolved%20through%20microsoft%20support.%20Apparently%20for%20some%20reason%20the%20OMI%20Agent%20was%20in%20a%20zombie%2Fstuck%20state.%20Restarting%20the%20agent%20didnt%20work%2C%20had%20to%20manually%20kill%20the%20process%20and%20start%20the%20agent%20again.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EKilling%20and%20starting%20the%20agent%20again%20resolved%20the%20issue.%20As%20per%20MS%20teams%20%2C%20one%20of%20the%20possibility%20of%20this%20behavior%20may%20be%20that%20the%20disk%20space%20got%20full%20earlier%20at%20some%20point%20in%20time%2C%20which%20was%20then%20resolved%20however%20may%20be%20that%20disk%20space%20issue%20might%20have%20caused%20the%20agent%20to%20go%20into%20such%20a%20state.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAnyways%2C%20the%20issue%20has%20been%20resolved%2C%20we%20are%20still%20monitoring%20to%20see%20if%20it%20remains%20stable%20or%20not.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESharing%20the%20above%20for%20the%20benefit%20of%20all.%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EThanks%20once%20again%20for%20your%20support%2C%20much%20appreciated.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFahad.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Contributor

Hello,

 

We have observed that we no longer are receiving Syslog and CEF logs from the Azure Sentinel Log forwarder that is deployed on client premise. I have performed the following steps:

 

netstat -an | grep 514

Status: Listening or established (which is fine)

 

netstat -an | grep 25226

Status: Listening or established (which is fine)

 

sudo tcpdump -A -ni any port 514 -vv

Status: receive logs from the data sources (which is fine)

 

sudo tcpdump -A -ni any port 514 -vv | grep (Zscaler IP)

Status: receive logs from the Zscaler data source, the logs showed Palo Alto name in the CEF messages which means Zscaler traffic was routed through the firewall (which is fine, as confirmed by client)

 

sudo tcpdump -A -ni any port 25226 -vv

Status: No logs were received (Issue Identified)

 

sudo tcpdump -A -ni any port 25226 -vv | grep (Zscaler IP)

Status: No logs were received (Issue Identified)

 

Restarted the Rsyslog Service:

service rsyslog restart (After service restart, Azure Sentinel Started receiving the syslog. The Syslog data source came up and working fine)

 

Restarted the OMSAgent Service

/opt/microsoft/omsagent/bin/service_control restart {workspace ID}

Status: There was no status message and prompt came, assuming it restarted in the background (Please confirm if this is the normal, not prompting or showing any message)

 

After OMS Agent restart, ran tcpdump again on OMS Agent to see if it starts receiving the logs but no luck.

 

I followed the following link: https://docs.microsoft.com/en-us/azure/sentinel/troubleshooting-cef-syslog?tabs=rsyslog

 

Can any one guide what probably be the cause of this issue????? Any help will be much appreciated. Thanks in advance.

2 Replies
I would also check with your Azure admins to see if they may have added any firewall rules that could have blocked the data. It seems that there is one command from the document to check for this:
sudo wget -O cef_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_troubleshoot.py... python cef_troubleshoot.py [WorkspaceID]

@Gary Bushey  Thank you for the prompt response. We have got the issue resolved through microsoft support. Apparently for some reason the OMI Agent was in a zombie/stuck state. Restarting the agent didnt work, had to manually kill the process and start the agent again. 

 

Killing and starting the agent again resolved the issue. As per MS teams , one of the possibility of this behavior may be that the disk space got full earlier at some point in time, which was then resolved however may be that disk space issue might have caused the agent to go into such a state.

 

Anyways, the issue has been resolved, we are still monitoring to see if it remains stable or not.

 

Sharing the above for the benefit of all.


Thanks once again for your support, much appreciated.

 

Fahad.