Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community

Infoblox and Parsing Questions

Copper Contributor

Hello,

 

Have Infoblox DNS Query/Response logs been tested with Azure Sentinel ?

I am testing it and have found that Infoblox DNS seems to generate only Threat Logs in CEF. The other DNS logging categories, such as DNS Queries/Responses, are logged in some non-CEF format over syslog, like the following:

##<166>Dec 23 12:54:05 infoblox1.localdomain named[12821]: client @0x7fbc3c0cc6e0 192.168.80.1#57296 (server1.fwd1): query: server1.fwd1 IN A + (192.168.80.200)

 

I can't even see these logs in the Sentinel Workspace. The logs arrive at the on-repm Syslog Agent and are forwarded to omsagent process over port 25226, but beyond that i don't see them anywhere.

The OMSAgent fluentd parsing checks that the incoming message has "CEF or ASA" keywords before processing the message further. Which seems to be a showstopper for the above mentioned syslog message.

 

Please advise:

1. Should we create a custom parser for Infoblox query/response logs or Microsoft has already addressed them  ?

2. Does the syslog message(payload) parsing occur at the OMSAgent side or at the Azure Sentinel Workspace side ?

 

3. By having a vendor connector listed in Azure Sentinel connector list, such as ASA, Fortigate, .., does this mean having "parser" in the background ? I noticed that vendor connectors do query the CommonSecurityLog with filter of "device vendor" , so i don't fully understand the technical meaning of "having a connector for X vendor".

 

4. How to troubleshoot logs processing and ingestion after the logs are delivered from the syslog daemon to the omsagent daemon? Any troublehsoot files or tables to look into  ?

 

Thanks in advance.

 

14 Replies

Hello @majo1 

 

I would suggest checking that you configure to receive from the correct Syslog Facility from Infoblox devices.
As there is no connector for Infoblox at the time being, it means that there are no pre-built queries, workbooks, notebooks that are already made by Microsoft inside Azure Sentinel. However, you can always look at the community GitHub to see if there is some work that has been made to enrich Infoblox logs.
If you have a connector for an existing solution such as for instance Palo Alto Networks or Fortinet, you can use pre-build queries (Kusto queries), Dashboards (Notebooks), ... that have already been pre-made for you.

Also, you could check at everything linked to the "DNS" connector as some of the hunting queries could be adapted to work with Infoblox logs.

Hope it helps,

Thomas

@thomasdefise 

 

Thanks Thomas.

I don't think "facility" has something to do with the case of infoblox query/response logs, because Fluentd settings match on two keywords in order to process logs further and those are CEF/ASA . Infoblox query/response logs doesn't have any of the two keywords.

 

I understand from you that A Sentinel Connector has nothing to do with parsing. Correct ?

Do you know where syslog payload parsing takes place ? At OMSAgent side or At Sentinel WA side ?

 

 

@majo1 According to my experience with Azure Sentinel, the parsing has to be done at the Syslog server.
However, I would imagine that there could be a trick to parse it using Azure Logic App or Azure Functions but would come with additional cost.
For your case, I would first check on the Syslog appliance if they Infoblox can send logs in the CEF format and if not parse the logs at the Syslog server and make sure they are in the CEF format which is an industry-standard log format on top of Syslog.

I on the Infoblox documentation that for instance "Threat Protection Events" can be sent in the CEF format. https://docs.infoblox.com/display/nios84/Monitoring+through+Syslog

Hope it helps.

hello, any luck on this one?
https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/Infoblox%20NIOS
This solution includes various parsers including DNS log Parser.

@varunkohli 

I've seen that - and the connector is indeed in Sentinel - InfoBlox NIOS (Preview). But neither InfoBlox, nor Microsoft describe in what fashion they expect the data to arrive in [from the documentation I've read]. I'm assuming given the poorly laid out logs, this is supposed to be a standardized syslog message. Can you confirm that?

 

I've defined within the security-config-omsagent.conf file a line that handle its specific syslog. It amounts to nothing more than a policy which checks if the hostname is contained in the raw payload. conf.pngTable.png

 

My problem here is that Sentinel refuses to recognize that InfoBlox NIOS logs are now flowing into the Syslog table. Attempts to manually add the Parser Functions through the Github link simply fail to execute (scalar problem with |project Source). Indeed, fi I do check the Syslog table with a DHCPD parser, I get results.

 

mredbourne2405_0-1666810064967.png

 

 

Which is easy enough to fix, if all that's truly doing is considering the data source - which in this case will always contain "infoblox". But I'm still faced with problems concerning the data connector, which would be preferable to have operational. Any insights into this?

I have been going back and forth with Microsoft support on this for months, I am experiencing the exact same issues as mredbourne2405. Has anyone found a solution or workaround?

@JasonS1990 

Hey Jason,

I have about 2 dozen (or so) Infoblox sub-parsers attached to a primary parser. "Infoblox" is the primary one, and unions the other ones together.

mredbourne2405_0-1676557824872.png

Here's one of the subparsers:

mredbourne2405_1-1676557863430.png

I would also double check your Watchlists defined in Sentinel. There should be a Watchlist called "Sources_by_SourceType". In it you need a SourceType called "InfobloxNIOS" with one or more keys assigned to it.

mredbourne2405_3-1676558085752.png

I set my both to the Hostname and the FQDN of the reporting log sources. (Some information scrubbed...)

 

Double check that those are set up correctly. If they are, attach a couple error messages from Sentinel so I can review it. We did eventually get NIOS logs working - though without support from Microsoft.

 

Hey thanks for getting back to me!

So I did try the watchlist solution before with support, I made the source type "infobloxNIOS" like the data connector seems to want and the source as our infoblox servers. Unfortunately it didn't seem to change anything. The data connector still hasn't changed to green indicating a connection..

as far as errors there is none, that's the odd thing. I am technically getting the data its just not parsing correctly. Can you share that sub parser code though? I have not tried that yet

@JasonS1990 Attached is a "CSV". It's a regular text file - MIcrosoft wouldn't let me upload a *.txt. That should be all of them. Have you looked at the Content Hub? They have these parsers (n=22) in there. Just make sure you completely remove the NIOS stuff before attempting to install it. Otherwise it'll attempt to create multiple objects with the same name that causes problems.

@mredbourne2405 thanks for your help so far! I tried what you mentioned and got the parsers installed but unfortunately they come back empty and infoblox data connector still shows as grey and not connected from the connector page. I have attached some screenshots let me know if this looks right to you. Also which linux disto were you using? The only other thing I can think of is were using AlmaLinux a bit for bit remake of CentOS but may be the culprit.

info2.PNGinfo1.PNGinfo3.pnginfo4.png

For the time being we're on CentOS 7 Linux (rsyslog configuration) with plans to migrate to RHEL 9.1 or a derivative of RHEL 9. However, our systems are not a good baseline to use as they're heavily modified to support large EPS ingestion (~30k EPS across 3 nodes being our heaviest. We have nearly 50 collectors deployed in total).

Run this on the server where your rsyslog/syslog-ng collector and post your output.
Command >> netstat -tunlp | egrep "25226|25224".

What protocol is 25224 and 25226 listening on? 25226 should come back TCP, 25224 should come back UDP (if default). If that's true, you need to modify your rsyslog configuration for the NIOS logs and use the line "if $rawmsg contains "vnios" [...] then @127.0.0.1:25224". @ means UDP, @@ means TCP.

The default configuration for the Syslog (used by NIOS) connector on any appliance is UDP transport (@127.0.0.1:25224). You can use TCP for Syslog as well, but that's requires modifying fluentd configurations and disabling some 'helper' processes on the OMS Agent. (It's also worth noting that such changes will put you out of scope for assistance from most Microsoft technicians for troubleshooting and isn't something I'd recommend doing unless you have deep knowledge of the software.)

As an aside to the rsyslog configuration, you're missing the "& stop" line after your CEF checks. That will hamper your performance.

@mredbourne2405 Ah ok that makes so much sense. It's odd that the Log Analytics agent installer script didn't do this during installation and the Infoblox NIOS data connector directions don't mention this at all... 

 

Anyway here is a screenshot of the output and the new config. let me know if this looks correct to you. Capture2.PNGCapture.PNG

A moment of candid honesty (I am NOT a Microsoft Employee, and this is one of my only complaints) - the installer (both for CEF and the generic OMS Agent) are both terrible. The performance on them leaves a lot to be desired. The installer doesn't really account for any other scenarios (Eg: ingesting Meraki logs, NIOS, etc). There's a whole other level of complexity in this that most people aren't even aware of yet.

Case and point: Ask the creators of "rsyslog" how expensive message searching is. Aside from ereregex or breregex, it's one of the most expensive methods to split traffic on. My larger servers use property search (eg: ':fromhost-ip, equals, "1.1.1.1" @/@@<location>'.

In our other managed collectors (legacy through another vendor - IBM - we could push 50k EPS on a single node if necessary and had the compute resources + licensing on hand. That's not possible with the OMS Agent without load balancing. The new product coming (AMA for Linux) from my testing and discussions with Microsoft Engineers is rumoured to be marginally better than my current setup. We'll move over once I have support for RHEL9.

That aside, the config looks good now. In Sentinel run the following commands:
Syslog
| where Computer contains "vnios" or Computer contains "ns1" or Computer contains "ns2"
| summarize count() by Computer

Does that return information? If so, run the Infoblox function:
Infoblox
| take 10