Forum Discussion
Ingestion of Custom Logs of Files (Never Updated) in Azure Sentinel
Hi all, I need to send the custom logs of a CSV file to Azure Sentinel. New files are written daily on a collector device from several applications running on other devices. I was able to transfer the files on the RedHat 7 collector device through SFTP but the logs are not sent to the Log Analytics Workspace of Sentinel by the Log Analytics Agent.
The Agent is correctly configured (custom log path, etc...) because I was able to receive a log when I manually updated one of the log files. The problem here is that the new file is never flushed with new data. From the documentation (https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs) I see this statement:
"Custom log collection requires that the application writing the log file flushes the log content to the disk periodically. This is because the custom log collection relies on filesystem change notifications for the log file being tracked."
I simply want the Log Analytics Agent for Linux to get the data from the newly created file (the file is never updated) and push all the logs to the Workspace. Is it possible with the Log Analytics Agent for Linux?
Thank you very much in advanced
Simone
11 Replies
- GaryBusheyBronze Contributor
simonepatonico Does the new file following the required naming standards in regards to having the date and time as part of the filename?
- simonepatonicoBrass Contributor
GaryBushey For now I did some tests with files named Norma01.csv, Norma02.csv, etc...
I did the configuration required on the Log Analytics Workspace as you can see from the attached figure.
- GaryBusheyBronze Contributor
simonepatonico Looking at the document link you posted it states the following. Are you following this naming convention? It did not look like it from your image unless there is only 1 entry in the file that was shown in the image.
The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry.
YYYY-MM-DD HH:MM:SS
M/D/YYYY HH:MM:SS AM/PM
Mon DD, YYYY HH:MM:SS
yyMMdd HH:mm:ss
ddMMyy HH:mm:ss
MMM d hh:mm:ss
dd/MMM/yyyy:HH:mm:ss zzz
yyyy-MM-ddTHH:mm:ssK