Azure ATP connection closed errors

[archedmeerkat]

I am seeing the following error in the Azure ATP Sensor logs in my environment when running net group "Domain Admins" /domain from member workstations. I do not see the correlated event of a user querying Domain Admins in AATP portal either. I am using VMWare to virtualize all components and have made the settings adjustments mentioned in the Azure ATP documentation for network adapters on VMWare hosted systems with the AATP sensor.

Has anyone else seen this? Can anyone from Microsoft provide some guidance? All other detections appear to be working as I only see the parsing errors when running net group or net user queries (I have not done exhaustive testing).

2019-06-21 18:20:37.0249 Error TransportStreamExtension+Disposable Error parsing segment [transportSessionKey=DCIP:445 => WorkstationIP:57133]
Microsoft.Tri.Infrastructure.ExtendedException: Parser read out of bounds [_messageLength=0 readDataLength=3]
at void Microsoft.Tri.Sensor.TransportStreamExtension+Disposable.Dispose()
at void Microsoft.Tri.Sensor.SmbParser.ParseDceRpcPayload(TransportPacket transportPacket, Session session, Command command, ITransportDataStream transportDataStream)
at void Microsoft.Tri.Sensor.SmbParser.ParseReadPayload(TransportPacket transportPacket, Session session, Command command, ITransportDataStream transportDataStream)
at bool Microsoft.Tri.Sensor.SmbParser.ParseSmb1Command(TransportPacket transportPacket, SessionId transportSessionId, Session session, Header header, PayloadInfo payloadInfo, bool isLastInChain, ITransportDataStream transportDataStream)
at void Microsoft.Tri.Sensor.SmbParser.ParseSmb1(TransportPacket transportPacket, SessionId transportSessionId, uint messageLength, ITransportDataStream transportDataStream)
at void Microsoft.Tri.Sensor.SmbParser.Parse(TransportPacket transportPacket, SessionId transportSessionId, uint messageLength, ITransportDataStream transportDataStream)
at void Microsoft.Tri.Sensor.TcpParser.ParseApplicationProtocol(IpDatagram ipDatagram, TransportSessionKey transportSessionKey, TcpSession tcpSession)
at void Microsoft.Tri.Sensor.TcpParser.ParseInternal(IpDatagram ipDatagram, TransportSessionKey transportSessionKey, BufferSlice bufferSlice)
at void Microsoft.Tri.Sensor.TcpParser.Parse(IpDatagram ipDatagram, BufferSlice bufferSlice)

[#2]

@archedmeerkat  Can you verify TSO offload is disabled?

from elevated powershell,  run:

Get-NetAdapterAdvancedProperty | Where-Object DisplayName -Match "^Large*"

Check it the feature is enabled, if it is, run:

Disable-NetAdapterLso -Name {name of adapter}  \\ this will disable LSO for both IPv4 and IPv6.

Then verify the previous command again to make sure it was disabled.

Eli

[#3]

@Eli Ofek- Commands returned that TSO offload is disabled on both on ipv4 and ipv6

[#4]

@Eli Ofek- Is there a way to enable Debug logging or extra logging on the ATP sensor? It appears to only be happening on one of the four sensors we have.

[#5]

@archedmeerkat Hi, I am on-boarding internally the engineer who wrote most of the code for parsing this protocol...

For now I don't think raising  the trace log will produce meaningful results.

But something that might help is if you are able to use netmon 3.4 to capture a cap file trace of this specific traffic (recording while we have at least one incident like this in the log)

in which case we can use it to repro the problem in our lab which will speed up the research considerably... 

[#6]

@Eli Ofek- Can I use built in netsh commands to run the trace or will it have to be with netmon 3.4.

Do you want both ends or just from the AATP Sensor?

[#7]

@archedmeerkat 

netmon is preferred. if there is no such option, we can try to work with netsh, but it will take more time.

we need to capture the traffic on the DC.

Also, it's interesting if you can get us a capture while invoking this command against the captured DC:

net group "Domain Admins" /domain

It seems you have some SMB1 traffic there, which is not very common these days, so we want to make sure our parser is not missing anything from this protocol which we don't get a lot any more.

Eli

[#8]

@Eli Ofek- I have the capture with net mon. Is there anywhere I can securely upload this or share it with you?

[#9]

@archedmeerkat , The best option is to open a support case and give me the case #.

I will ask the assigned engineer to open a secured workspace for you where we can exchange files,

And also to add me to the case thread so I can help and add PG members as needed.

[#10]

@Eli Ofek

I've sent the case number over in a private message, but wasn't sure how to add you to the case. I'll see if I can have that done shortly.

[#11 · Accepted solution]

@archedmeerkat 

Engineering has researched the sampled capture ans managed to reproduce the issue.

Sadly, this is not an easy fix, it's a specific traffic/rare traffic on top of SMB1 we were not aware of before and currently cannot parse.

We have opened a bug for it.

It is planned but in low priority for now as telemetry shows it happens rarely.

We will update once we get it resolved so the fix can be verified.

[#12]

@Eli Ofek 

Have only seen this in our lab so far, so I think the impact is currently pretty low for us.

[#13]

@archedmeerkat , Yes, I figured so as telemetry showed it is rare as well, I assume not too many people use protocols on top of SMB1 anymore which is good :)