Forum Discussion
OMS Agent on Azure Sentinel Log forwarder not receiving and forwarding logs to sentinel workspace
GaryBushey Thank you for the prompt response. We have got the issue resolved through microsoft support. Apparently for some reason the OMI Agent was in a zombie/stuck state. Restarting the agent didnt work, had to manually kill the process and start the agent again.
Killing and starting the agent again resolved the issue. As per MS teams , one of the possibility of this behavior may be that the disk space got full earlier at some point in time, which was then resolved however may be that disk space issue might have caused the agent to go into such a state.
Anyways, the issue has been resolved, we are still monitoring to see if it remains stable or not.
Sharing the above for the benefit of all.
Thanks once again for your support, much appreciated.
Fahad.
- KennethMLApr 14, 2023MCT
Hi alazarg
Check your /etc/rsyslog.d/50-default.conf file. It has a section like this:
# # First some standard log files. Log by facility. # #auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log #daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log #lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log #user.* -/var/log/user.logThe problem is line 5 where any facility (*.*) is written to /var/log/syslog file. Solution is to either comment it out (using #) or even better:
# # First some standard log files. Log by facility. # if ($fromhost-ip == '127.0.0.1') then { #auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log #daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log #lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log #user.* -/var/log/user.log }Then only local generated logs will be written to the filesystem and the disk will not be filled.
Good luck.
/Kenneth ML
- alazargMay 02, 2023Copper Contributor
Thank you very much for the input KennethML
The disk size full issue was resolved by increasing the OS Disk size. When creating Linux VM in Azure, OS disk is 32GB and for Log forwarder that need to be increased to at least 256 GB, as follows:
Select the disk, choose from the list, and save.
The syslog server will maintain the log file cleanup with its scheduled task.