Sentinel alerts stopped running playbooks

%3CLINGO-SUB%20id%3D%22lingo-sub-1087210%22%20slang%3D%22en-US%22%3ESentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087210%22%20slang%3D%22en-US%22%3E%3CP%3EI%20have%20at%20least%20four%20instances%20of%20Sentinel%20where%20the%20alerts%20create%20the%20incidents%20but%20don't%20run%20the%20associated%20playbooks.%20This%20seemed%20to%20have%20started%20somewhere%20around%20Dec%2030th.%20There%20are%20no%20failed%20runs%20for%20the%20logic%20apps%2C%20and%20if%20I%20trigger%20the%20playbook%20from%20the%20incident%20detailed%20view%2C%20it%20works%20without%20any%20problem.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20scheduled%20logic%20apps%20(using%20the%20Recurrence%20trigger)%20and%20they%20work%20fine%20but%20those%20that%20are%20supposed%20to%20be%20triggered%20by%20an%20Azure%20Sentinel%20alert%20are%20not%20running%20even%20though%20there%20are%20alerts%20raised.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20opened%20a%20ticket%20with%20Microsoft%20but%20I%20didn't%20receive%20any%20reply%20so%20far.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERegards%2C%3C%2FP%3E%3CP%3EAdrian%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087315%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087315%22%20slang%3D%22en-US%22%3E%3CP%3EAlso%20having%20the%20same%20issue%20across%203%20tenants.%20Problem%20started%20for%20us%20around%209am%20EST%20on%20the%2031st.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20can%20run%20them%20manually%20within%20the%20Incident%20details%20page%2C%20but%20triggers%20are%20failing%20if%20I%20run%20them%20in%20the%20Logic%20App%20page.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAfter%20some%20digging%20around%20in%20the%20logic%20app%20code%2C%20I%20looked%20at%20the%20raw%20output%20of%20the%20block%20that's%20failing%20I%20found%20that%20the%20header%20is%20not%20populating%20correctly%20and%20the%20body%20is%20not%20populating%20at%20all.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087351%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087351%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3BYou're%20probably%20right.%20That's%20probably%20why%20the%20raw%20output%20of%20the%20trigger%20block%20isn't%20populating%20properly.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087336%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087336%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F418279%22%20target%3D%22_blank%22%3E%40leoszalkowski%3C%2FA%3E%26nbsp%3BI've%20seen%20your%20post%20and%20the%20problem%20looks%20quite%20similar.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20playbook%20would%20not%20work%20if%20one%20triggers%20the%20%22Sentinel%20Alert%22%20manually%20because%20is%20missing%20the%20data%20from%20the%20alert%20itself.%20For%20this%20reason%2C%20when%20used%20from%20the%20Incident%20details%20interface%2C%20the%20playbook%20works%20because%20it%20is%20receiving%20the%20alert%20details.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20don't%20think%20this%20is%20a%20problem%20with%20the%20playbooks%20as%20they%20are%20not%20showing%20with%20failed%20runs.%20Most%20likely%20is%20an%20issue%20with%20the%20Azure%20Sentinel%20Logic%20App%20trigger%20(that's%20still%20in%20Preview%20mode).%20I%20will%20create%20a%20new%20playbook%20from%20scratch%20and%20see%20if%20it%20makes%20any%20difference.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087446%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087446%22%20slang%3D%22en-US%22%3EI%20got%20the%20same%20issue%20and%20raised%20a%20ticket%20on%2030%2F12.%20Triggering%20the%20playbooks%20manually%20from%20the%20incidents%20works%20as%20a%20work%20around.%20Been%20in%20touch%20with%20the%20support%20just%20now%20and%20it%20seems%20to%20be%20a%20general%20issue%20thats%20being%20worked%20on%20so%20MS%20is%20aware%20and%20working%20on%20it.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087570%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087570%22%20slang%3D%22en-US%22%3E%3CP%3EAn%20interesting%20thing%2C%20I%20created%20a%20dummy%20playbook%2C%20assigned%20it%20to%20the%20alert%20and%20it%20worked.%20I%20switched%20back%20to%20the%20original%20playbook%20and%20now%20the%20alert%20triggers%20it.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087659%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087659%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3BThat's%20odd.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESo%20you%20assigned%20the%20alert%20the%20new%20dummy%20playbook%20and%20reassigned%20the%20alert%20the%20old%20playbook%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1091189%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1091189%22%20slang%3D%22en-US%22%3EGood%20for%20you%20but%20strange%20for%20me.%20I%20tried%20that%20aswell.%20Twice.%20Also%20creating%20a%20new%20playbook%20from%20scratch%20And%20assigning%20but%20the%20result%20was%20the%20same.%20Maybe%20something%20have%20changed%20and%20I%20should%20try%20again.%20That%20or%20I%E2%80%99m%20missing%20something.%20But%20second%20time%20around%20I%20had%20me%20support%20in%20session%20so%20it%20shouldn%E2%80%99t%20be.%3CBR%20%2F%3E%3CBR%20%2F%3EAnyone%20know%20if%20there%20is%20a%20place%20where%20ongoing%20issues%20are%20published%3F%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1087942%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1087942%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F418279%22%20target%3D%22_blank%22%3E%40leoszalkowski%3C%2FA%3E%26nbsp%3BThat's%20correct%2C%20I%20assigned%20a%20new%20playbook%2C%20saved%2C%20reassigned%20the%20old%20playbook%20and%20it%20worked%20after%20that.%26nbsp%3B%20In%20fact%20you%20can%20just%20remove%20the%20existing%20playbook%2C%20save%2C%20and%20then%20reassign%2C%20no%20need%20for%20a%20%22temporary%22%26nbsp%3B%20one.%20The%20problem%20is%20that%20I%20would%20have%20to%20do%20that%20for%20every%20alert%20configured.%20We%20have%20hundreds%20of%20them%20so%20I%20would%20rather%20not%20go%20that%20path.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1289700%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1289700%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHaving%20the%20exact%20same%20issues%2C%20has%20there%20been%20any%20progress%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ETried%20the%20workaround%20you%20suggested%20but%20no%20success%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%3C%2FP%3E%3CP%3ENeil%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1290366%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1290366%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F296852%22%20target%3D%22_blank%22%3E%40Neil2020%3C%2FA%3E%26nbsp%3BIt%20just%20fixed%20by%20itself%20after%20a%20couple%20of%20days%2C%20we%20didn't%20have%20to%20do%20anything.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1292086%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1292086%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWow%2C%20still%20broken%20for%20me%20so%20raised%20a%20suport%20case%2C%20they%20have%20said%20it%20is%20being%20escalated%20so%20I%20will%20wait%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20responding%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1344950%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1344950%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F296852%22%20target%3D%22_blank%22%3E%40Neil2020%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHave%20you%20heard%20anything%3F%26nbsp%3B%20I'm%20have%20the%20same%20problem.%20%26nbsp%3B%20The%20playbook%20runs%20manually%20from%20sentinel%20incidents%20page%20but%20doesn't%20trigger%20on%20new%20alerts.%26nbsp%3B%20I%20need%20it%20to%20trigger%20since%20this%20logic%20app%20is%20for%20notification%20of%20new%20incidents.%20%26nbsp%3B%26nbsp%3B%20Any%20insight%20would%20be%20appreciated%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1345083%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1345083%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F574633%22%20target%3D%22_blank%22%3E%40Secuerskydev%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAs%20a%20test%2C%20I%20suggest%20that%20you%20delete%20and%20recreate%20the%20alert%20to%20see%20if%20it%20makes%20any%20difference.%20In%20some%20situations%20it%20appears%20that%20the%20%22sync%22%20between%20the%20alert%20and%20the%20playbook%20(aka%20an%20%22action%22)%20is%20lost%20or%20misconfigured%20so%20you%20may%20have%20a%20situation%20where%20an%20alert%20may%20look%20like%20is%20assigned%20to%20a%20playbook%20but%20in%20reality%20is%20not.%20This%20could%20also%20cause%20the%20opposite%20of%20not%20running%20playbooks%2C%20when%20the%20playbook%20is%20ran%20several%20times.%20That%20again%20we%20found%20out%20was%20due%20to%20the%20alert%20having%20several%20%22actions%22%20for%20the%20same%20playbook%20(the%20Sentinel%20%22actions%22%20are%20only%20accessible%20throught%20the%20API).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAdrian%20Grigorof%3C%2FP%3E%3CP%3E%3CA%20href%3D%22http%3A%2F%2Fwww.managedsentinel.com%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ewww.managedsentinel.com%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1406907%22%20slang%3D%22en-US%22%3ERe%3A%20Sentinel%20alerts%20stopped%20running%20playbooks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1406907%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F353788%22%20target%3D%22_blank%22%3E%40AdiGrio%3C%2FA%3E%2C%20Thanks%20that%20worked!%3C%2FP%3E%3C%2FLINGO-BODY%3E
Contributor

I have at least four instances of Sentinel where the alerts create the incidents but don't run the associated playbooks. This seemed to have started somewhere around Dec 30th. There are no failed runs for the logic apps, and if I trigger the playbook from the incident detailed view, it works without any problem. 

 

I have scheduled logic apps (using the Recurrence trigger) and they work fine but those that are supposed to be triggered by an Azure Sentinel alert are not running even though there are alerts raised.

 

I opened a ticket with Microsoft but I didn't receive any reply so far.

 

Regards,

Adrian

14 Replies

Also having the same issue across 3 tenants. Problem started for us around 9am EST on the 31st. 

 

I can run them manually within the Incident details page, but triggers are failing if I run them in the Logic App page.

 

After some digging around in the logic app code, I looked at the raw output of the block that's failing I found that the header is not populating correctly and the body is not populating at all.

@leoszalkowski I've seen your post and the problem looks quite similar. 

 

The playbook would not work if one triggers the "Sentinel Alert" manually because is missing the data from the alert itself. For this reason, when used from the Incident details interface, the playbook works because it is receiving the alert details.

 

I don't think this is a problem with the playbooks as they are not showing with failed runs. Most likely is an issue with the Azure Sentinel Logic App trigger (that's still in Preview mode). I will create a new playbook from scratch and see if it makes any difference.

@AdiGrio You're probably right. That's probably why the raw output of the trigger block isn't populating properly. 

 

 

I got the same issue and raised a ticket on 30/12. Triggering the playbooks manually from the incidents works as a work around. Been in touch with the support just now and it seems to be a general issue thats being worked on so MS is aware and working on it.

An interesting thing, I created a dummy playbook, assigned it to the alert and it worked. I switched back to the original playbook and now the alert triggers it.

@AdiGrio That's odd. 

 

So you assigned the alert the new dummy playbook and reassigned the alert the old playbook?

@leoszalkowski That's correct, I assigned a new playbook, saved, reassigned the old playbook and it worked after that.  In fact you can just remove the existing playbook, save, and then reassign, no need for a "temporary"  one. The problem is that I would have to do that for every alert configured. We have hundreds of them so I would rather not go that path.

Good for you but strange for me. I tried that aswell. Twice. Also creating a new playbook from scratch And assigning but the result was the same. Maybe something have changed and I should try again. That or I’m missing something. But second time around I had me support in session so it shouldn’t be.

Anyone know if there is a place where ongoing issues are published?

@AdiGrio 

 

Having the exact same issues, has there been any progress?

 

Tried the workaround you suggested but no success,

 

Thanks

Neil

@Neil2020 It just fixed by itself after a couple of days, we didn't have to do anything.

@AdiGrio 

 

Wow, still broken for me so raised a suport case, they have said it is being escalated so I will wait,

 

Thanks for responding

@Neil2020 

 

Have you heard anything?  I'm have the same problem.   The playbook runs manually from sentinel incidents page but doesn't trigger on new alerts.  I need it to trigger since this logic app is for notification of new incidents.    Any insight would be appreciated 

@Secuerskydev 

 

As a test, I suggest that you delete and recreate the alert to see if it makes any difference. In some situations it appears that the "sync" between the alert and the playbook (aka an "action") is lost or misconfigured so you may have a situation where an alert may look like is assigned to a playbook but in reality is not. This could also cause the opposite of not running playbooks, when the playbook is ran several times. That again we found out was due to the alert having several "actions" for the same playbook (the Sentinel "actions" are only accessible throught the API).

 

Adrian Grigorof

www.managedsentinel.com

@AdiGrio, Thanks that worked!