Last week, I on-boarded my DNS servers into the Sentinel DNS Connector (preview). Initially, things were working great, I saw DNS operational activity like Registrations, Config changes and (the ones I was most interested in) LookupQueries.
After a few hours, though, one of my DNS servers stopped reporting queries. It continued to send the registrations and config changes, but I saw no more queries for a few hours - then it resumed.
I'm confident that the server was still handling queries during that time. We have a load balancing solution in front of two nodes. The other node reported queries, the LB Config has not changed, and the LB sees both nodes as available.
If that had been the only blip, no problem. But since then, out of my three DNS servers, I'm rarely getting query data back from all three at the same time and most of the time I'm getting zero to one server reporting. They do seem to self-heal, but I can wake up the monitoring agent by using the DNS solution configure panel, tweaking the 'Talkative Client Threshold' and saving, but that doesn't last very long.
Looking at the 'Operations Manager' event logs on one of these servers, I see events 4512 and 1103 that have contents like:
4512: Converting data batch to XML failed with error "0x80131500" (0x80131500) in rule "Microsoft.SystemCenter.CollectDnsEtwEvents" running for instance "" with id:"{<guid>}" in management group "AOI-<guid>".
1103: Summary: 1 rule(s)/monitor(s) failed and got unloaded, 0 of them reached the failure limit that prevents automatic reload. Management group "AOI-<guid>". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).
After a few of those, it looks like the monitor job stops and doesn't restart.
I'm not sure where to find details on the rule ID or the record that is failing to parse, but it looks like something is failing to convert and after a few runs the whole system gives up trying.
Where do I go from here?
