SOLVED
Home

Azure ATP sensor update and communication error

%3CLINGO-SUB%20id%3D%22lingo-sub-1143076%22%20slang%3D%22en-US%22%3EAzure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1143076%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3EI%20have%20noticed%20some%20errors%20on%20our%20ATP%20Health%20Center.%3C%2FP%3E%3CP%3EThe%20sensors%20installed%20on%20two%20DC%20randomly%20stopped%20communicating.%3C%2FP%3E%3CP%3EAfter%20some%20time%20the%20health%20alert%20is%20automatically%20closed.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EConcurrently%20with%20this%20errors%20I%20noticed%20on%20the%20sensor%20logs%20this%20entries%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E%3CSTRONG%3EMicrosoft.Tri.Sensor.Updater.log%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E2020-01-31%2002%3A41%3A55.5375%20Warn%26nbsp%3B%20ResourceManager%20RestrictCpuAsync%20process%20doesn't%20exist%20%5BProcess%3DMicrosoft.Tri.Sensor%5D%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E%3CSTRONG%3EMicrosoft.Tri.Sensor.log%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E2020-01-31%2002%3A41%3A34.4173%20Error%20FrameReader%601%20CaptureFrames%20exception%2C%20exiting%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3EMicrosoft.Tri.Sensor.FrameReaderException%3A%20Failed%20reading%20frame%20%5BresultCode%3D-1%20message%3Dread%20error%3A%20PacketReceivePacket%20failed%5D%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E%26nbsp%3B%26nbsp%3B%20at%20bool%20Microsoft.Tri.Sensor.FrameReader%3CTCAPTUREDEVICE%3E.TryReadFrame(out%20DateTime%20time%2C%20out%20BufferSlice%20bufferSlice)%3C%2FTCAPTUREDEVICE%3E%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E%26nbsp%3B%26nbsp%3B%20at%20bool%20Microsoft.Tri.Sensor.NetworkListener.ParseFrame(FrameReader%20frameReader)%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E%26nbsp%3B%26nbsp%3B%20at%20void%20Microsoft.Tri.Sensor.NetworkListener.CaptureFrames(LiveFrameReader%5B%5D%20liveFrameReaders)%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20event%20ID%207031%20is%20written%20on%20the%20System%20Event%20Log%3A%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3EThe%20Azure%20Advanced%20Threat%20Protection%20Sensor%20service%20terminated%20unexpectedly.%20It%20has%20done%20this%201%20time(s).%20The%20following%20corrective%20action%20will%20be%20taken%20in%205000%20milliseconds%3A%20Restart%20the%20service.%3C%2FFONT%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20sensor%20version%20is%20now%20(Jan%2C%2031th)%26nbsp%3B%3CSPAN%20class%3D%22%22%3E2.106.7618%20and%20is%20marked%20as%20up%20to%20date%20but%20the%20version%202.107%20is%20our%20from%20the%20Jan%2026th.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22%22%3EDoes%20anyone%20have%20any%20suggestion%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EThanks.%3C%2FP%3E%3CP%3EMike%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1143076%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EATP%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESensor%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EUpdates%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1144773%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1144773%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F227410%22%20target%3D%22_blank%22%3E%40Michele%20D'Angelantonio%3C%2FA%3E%26nbsp%3B%2C%3C%2FP%3E%0A%3CP%3EDid%20this%20sensor%20ever%20worked%2C%20or%20did%20it%20stopped%20working%20at%20some%20point%3F%3C%2FP%3E%0A%3CP%3Eis%20the%20message%20%22%3CSPAN%3ECaptureFrames%20exception%2C%20exiting%22%20logged%20after%20every%20start%20crash%20%3F%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EDi%20dyou%20recently%20installed%20any%20product%20that%20is%20using%20winpcap%20or%20npcap%20on%20the%20same%20machine%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1147476%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1147476%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F106935%22%20target%3D%22_blank%22%3E%40Eli%20Ofek%3C%2FA%3E%26nbsp%3Bthanks%20for%20your%20attention.%3C%2FP%3E%3CP%3EThe%20sensor%20worked%20on%20all%20DCs%20for%20some%20months.%20I%20have%20the%20first%20error%20on%20january%20(sensors%20installed%20on%20june).%3C%2FP%3E%3CP%3ENothing%20is%20changed%20on%20the%20DCs%2C%20the%20sowtware%20installed%20are%3A%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F168556iB35157E9E163FD04%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22clipboard_image_0.png%22%20title%3D%22clipboard_image_0.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3Efor%20OfficeSscan%20we%20did%20not%20use%20the%20firewall.%26nbsp%3B%20So%20I%20don't%20think%20there%20is%20something%20using%20npcap%20or%20winpcap.%3C%2FP%3E%3CP%3E%3CSPAN%3Ethe%20message%20%22%3C%2FSPAN%3E%3CSPAN%3ECaptureFrames%20exception%2C%20exiting%22%20is%20logged%20on%20every%20crash%20(on%20my%20log%20I%20have%20only%20the%20last%20two)%20but%20not%20at%20the%20exact%20time.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3Ethanks%20again%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EMike%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1147480%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1147480%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F227410%22%20target%3D%22_blank%22%3E%40Michele%20D'Angelantonio%3C%2FA%3E%26nbsp%3Bwas%20there%20any%20recent%20change%20to%20the%20network%20stack%3F%3C%2FP%3E%0A%3CP%3Enics%20removed%2Fadded%20%3F%20drivers%20changed%3F%3C%2FP%3E%0A%3CP%3Eis%20this%20a%20VM%20or%20a%20physical%20machine%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1147583%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1147583%22%20slang%3D%22en-US%22%3E%3CP%3Ehi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F106935%22%20target%3D%22_blank%22%3E%40Eli%20Ofek%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20verified%20again%20every%20log%20and%20I%20found%20the%20event%20ID%207031%20with%20the%20sensor%20restart%20even%20on%20june.%3C%2FP%3E%3CP%3Ethe%20exact%20error%20I%20coan%20find%20on%20the%20log%20is%3A%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3E2020-01-30%2002%3A17%3A01.9748%20Error%20FrameReader%601%20CaptureFrames%20exception%2C%20exiting%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3EMicrosoft.Tri.Sensor.FrameReaderException%3A%20Failed%20reading%20frame%20%5BresultCode%3D-1%20message%3Dread%20error%3A%20PacketReceivePacket%20failed%5D%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3Eat%20bool%20Microsoft.Tri.Sensor.FrameReader%3CTCAPTUREDEVICE%3E.TryReadFrame(out%20DateTime%20time%2C%20out%20BufferSlice%20bufferSlice)%3C%2FTCAPTUREDEVICE%3E%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3Eat%20bool%20Microsoft.Tri.Sensor.NetworkListener.ParseFrame(FrameReader%20frameReader)%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22andale%20mono%2Ctimes%22%20size%3D%222%22%3Eat%20void%20Microsoft.Tri.Sensor.NetworkListener.CaptureFrames(LiveFrameReader%5B%5D%20liveFrameReaders)%3C%2FFONT%3E%3C%2FP%3E%3CP%3ETwo%20DCs%20are%20VM%20on%20Hyper-V%20cluster%2C%20the%20third%20is%20on%20Azure.%20No%20network%20changes.%3C%2FP%3E%3CP%3EThe%20issue%20seems%20to%20happen%20completly%20randomly.%26nbsp%3B%3C%2FP%3E%3CP%3EMy%20first%20impression%20was%20that%20the%20issue%20could%20be%20connected%20to%20the%20sensor%20updates%20but%20I've%20no%20evidence%20of%20that.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI've%20seen%20that%20the%20new%20version%20of%20the%20sensor%20is%20available%20from%20the%20Jan%2026th%20but%20no%20DCs%20are%20updated%20(the%20ATP%20portal%20marks%20all%20three%20DCs%20as%20up%20to%20date%2C%20with%20version%202.106.7618).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1147609%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1147609%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F227410%22%20target%3D%22_blank%22%3E%40Michele%20D'Angelantonio%3C%2FA%3E%26nbsp%3B%202.107%20will%20be%20deployed%20in%20the%20coming%20days.%3C%2FP%3E%0A%3CP%3ESo%20just%20so%20I%20understand%20it%20better%20-%20it's%20not%20always%20crashing%2C%20but%20from%20time%20to%20time%20it%20crashes%2C%20and%20when%20it%20does%2C%20it's%20with%20this%20error%3F%3C%2FP%3E%0A%3CP%3EHow%20many%20total%20failures%26nbsp%3B%20since%20first%20installed%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20would%20advise%20to%20open%20a%20support%20ticket%20for%20this.%3C%2FP%3E%0A%3CP%3ESince%20we%20haven't%20seen%20it%20before%2C%20we%20will%20need%20to%20collect%20more%20data%20to%20analyze.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAlso%2C%20another%20thing%20you%20can%20try%20%2C%20is%20that%20if%20you%20currently%20work%20with%20the%20default%20winpcap%20driver%2C%20you%20can%20replace%20it%20with%20npcap%20and%20see%20if%20it%20resolves%20the%20issue.%3C%2FP%3E%0A%3CP%3ESee%20this%20for%20instructions%3A%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure-advanced-threat-protection%2Ftroubleshooting-atp-known-issues%23azure-atp-sensor-nic-teaming-issue-%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure-advanced-threat-protection%2Ftroubleshooting-atp-known-issues%23azure-atp-sensor-nic-teaming-issue-%3C%2FA%3E%3C%2FP%3E%0A%3CP%3Eyou%20can%20follow%20it%20even%20of%20not%20using%20nic%20teaming.%3C%2FP%3E%0A%3CP%3EJust%20make%20sure%20to%20use%20version%26nbsp%3B%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fnmap.org%2Fnpcap%2Fdist%2Fnpcap-0.9984.exe%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fnmap.org%2Fnpcap%2Fdist%2Fnpcap-0.9984.exe%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EAs%20we%20currently%20have%20compatibility%20issues%20with%20the%20newer%20version%20which%20we%20haven't%20fixed%20yet.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1186639%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20ATP%20sensor%20update%20and%20communication%20error%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1186639%22%20slang%3D%22en-US%22%3E%3CP%3Ethanks%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F106935%22%20target%3D%22_blank%22%3E%40Eli%20Ofek%3C%2FA%3E%20for%20your%20support%2C%26nbsp%3Bwe%20will%20probably%20open%20a%20ticket%20in%20the%20next%20days.%26nbsp%3B%20%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
New Contributor

Hello,

I have noticed some errors on our ATP Health Center.

The sensors installed on two DC randomly stopped communicating.

After some time the health alert is automatically closed.

 

Concurrently with this errors I noticed on the sensor logs this entries:

 

Microsoft.Tri.Sensor.Updater.log

2020-01-31 02:41:55.5375 Warn  ResourceManager RestrictCpuAsync process doesn't exist [Process=Microsoft.Tri.Sensor]

 

Microsoft.Tri.Sensor.log

2020-01-31 02:41:34.4173 Error FrameReader`1 CaptureFrames exception, exiting

Microsoft.Tri.Sensor.FrameReaderException: Failed reading frame [resultCode=-1 message=read error: PacketReceivePacket failed]

   at bool Microsoft.Tri.Sensor.FrameReader<TCaptureDevice>.TryReadFrame(out DateTime time, out BufferSlice bufferSlice)

   at bool Microsoft.Tri.Sensor.NetworkListener.ParseFrame(FrameReader frameReader)

   at void Microsoft.Tri.Sensor.NetworkListener.CaptureFrames(LiveFrameReader[] liveFrameReaders)

 

The event ID 7031 is written on the System Event Log:

The Azure Advanced Threat Protection Sensor service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.

 

The sensor version is now (Jan, 31th) 2.106.7618 and is marked as up to date but the version 2.107 is our from the Jan 26th.

 

Does anyone have any suggestion?

Thanks.

Mike

 

6 Replies
Highlighted

@Michele D'Angelantonio ,

Did this sensor ever worked, or did it stopped working at some point?

is the message "CaptureFrames exception, exiting" logged after every start crash ?

Di dyou recently installed any product that is using winpcap or npcap on the same machine?

Highlighted

@Eli Ofek thanks for your attention.

The sensor worked on all DCs for some months. I have the first error on january (sensors installed on june).

Nothing is changed on the DCs, the sowtware installed are:

clipboard_image_0.png

for OfficeSscan we did not use the firewall.  So I don't think there is something using npcap or winpcap.

the message "CaptureFrames exception, exiting" is logged on every crash (on my log I have only the last two) but not at the exact time.

thanks again

Mike

 

Highlighted

@Michele D'Angelantonio was there any recent change to the network stack?

nics removed/added ? drivers changed?

is this a VM or a physical machine?

Highlighted

hi @Eli Ofek 

I verified again every log and I found the event ID 7031 with the sensor restart even on june.

the exact error I coan find on the log is:

2020-01-30 02:17:01.9748 Error FrameReader`1 CaptureFrames exception, exiting
Microsoft.Tri.Sensor.FrameReaderException: Failed reading frame [resultCode=-1 message=read error: PacketReceivePacket failed]
at bool Microsoft.Tri.Sensor.FrameReader<TCaptureDevice>.TryReadFrame(out DateTime time, out BufferSlice bufferSlice)
at bool Microsoft.Tri.Sensor.NetworkListener.ParseFrame(FrameReader frameReader)
at void Microsoft.Tri.Sensor.NetworkListener.CaptureFrames(LiveFrameReader[] liveFrameReaders)

Two DCs are VM on Hyper-V cluster, the third is on Azure. No network changes.

The issue seems to happen completly randomly. 

My first impression was that the issue could be connected to the sensor updates but I've no evidence of that.

 

I've seen that the new version of the sensor is available from the Jan 26th but no DCs are updated (the ATP portal marks all three DCs as up to date, with version 2.106.7618).

 

 

Highlighted
Solution

@Michele D'Angelantonio  2.107 will be deployed in the coming days.

So just so I understand it better - it's not always crashing, but from time to time it crashes, and when it does, it's with this error?

How many total failures  since first installed?

 

I would advise to open a support ticket for this.

Since we haven't seen it before, we will need to collect more data to analyze.

 

Also, another thing you can try , is that if you currently work with the default winpcap driver, you can replace it with npcap and see if it resolves the issue.

See this for instructions:

https://docs.microsoft.com/en-us/azure-advanced-threat-protection/troubleshooting-atp-known-issues#a...

you can follow it even of not using nic teaming.

Just make sure to use version 
https://nmap.org/npcap/dist/npcap-0.9984.exe

As we currently have compatibility issues with the newer version which we haven't fixed yet.

Highlighted

thanks @Eli Ofek for your support, we will probably open a ticket in the next days.