Microsoft Sentinel is a cloud-native SIEM, enriched with AI and automation to provide expansive visibility across your digital environment.

When evaluating various solutions, your peers value hearing from people like you who’ve used the product. Review Microsoft Sentinel by filling out a Gartner Peer Insights survey and receive a $25 USD gift card (for customers only). Here are the Privacy/Guideline links: Microsoft Privacy Statement, Gartner’s Community Guidelines & Gartner Peer Insights Review Guide.

Microsoft Sentinel playbooks use Azure Logic Apps behind the scenes to perform workflow automation.  While they are powerful, they can be hard to debug when things go wrong.  This blog post will show some steps you can take to debug your playbooks.

This article assumes knowledge of how to create Microsoft Sentinel Analytic Rules, Automation Rules, and playbooks.

Create the playbook

You will want to create the playbook first, since one will need to be created before it can be used in the Automation rule.   In the example shown below, the playbook will iterate through all the alerts in the incident and generate a task based on information in that alert.

Keep in mind that you will need to grant the Managed Identity access to the Microsoft Sentinel instance so that it can add the task to the incident.

Create sample Analytic rule

When you are creating a new playbook, you will need to test it.  Rather than waiting for the perfect incident to be created, you can create a dummy Analytic rule that utilizes the KQL \"datatable\" operator to generate the information you need to test your playbook.

An example of this is shown below.  This code sample was taken from the \"datatable\" operator's documentation located at datatable operator - Azure Data Explorer & Real-Time Analytics | Microsoft Learn

let BlogTable = datatable (Date:datetime, Event:string, MoreData:dynamic)[\n    datetime(1910-06-11), \"Born\", dynamic({\"key1\":\"value1\", \"key2\":\"value2\"}),\n    datetime(1930-01-01), \"Enters Ecole Navale\", dynamic({\"key1\":\"value3\", \"key2\":\"value4\"}),\n    datetime(1953-01-01), \"Published first book\", dynamic({\"key1\":\"value5\", \"key2\":\"value6\"}),\n    datetime(1997-06-25), \"Died\", dynamic({\"key1\":\"value7\", \"key2\":\"value8\"}),\n];\nBlogTable

To trigger the loop in the playbook, set the \"Event Grouping\" to \"Trigger an alert for each event\" and enable \"Alert grouping\" and set it to \"Grouping all alerts trigger by this rule into a single incident\"

Create the Automation Rule

Create a new automation rule that triggers off the analytic rule that was created above.  This will make sure no other incidents will trigger the automation rule and playbook accidently while testing.   Once the playbook has been debugged, this automation rule can be modified to trigger off any other Analytic rules or all of them.

Set the \"Actions\" to run the playbook that was created above and save it.

Debugging the playbook

Once your analytic rule has triggered, you can disable it so that it does not create an excessive number of incidents.  This blog will show how to rerun a playbook with old data.

In the Microsoft Sentinel Automation Rules, select the \"Active playbooks\" tab and select your playbook.  You will see a page that look likes the image below showing how many runs have been performed and the status.   Note that in the image below, this playbook was run and had errors.   If it was successful, the \"status\" field will be set to \"Succeeded\"

Click on the identifier link to be brought into the playbook's history while will show the error. 

The full functionality of playbooks is outside the scope of this blog post, but note that by clicking the arrows in the loop, you can progress to the next error or the next value in the loop.  This is very useful if only certain entries in the loop are causing issues.

In this case, the error shown is \"BadRequest\".  This is a fairly generic error message but usually means that one of the parameters being set is wrong.  If you click on the step, a new pane will open showing all the information about the error including the inputs to the step and the output.  Use the \"INPUTS\" section to verify what parameters are being used and the \"OUTPUTS\" section to see what has been returned. 

Looking at the \"INPUTS\", shown below, doesn't provide too much useful information in this case.  It appears that an ARM Id is being passed in as \"incidentArmId\" and the \"taskTitle\" and \"taskDescription\" both have values.

Looking at the \"OUTPUTS\" section, shown below, provides a bit more useful information.  The \"innerError\" message states that \"Incident Arm id is invalid\" which provides a good place to start looking.   

To edit the playbook, close this run history page, and then in the \"Overview\" page, click the \"Edit\" button.

Click on the \"Add task to incident\" step to see what values are used.   This will open the parameters pane to see what parameters are being used as shown below

Notice that the \"Alert ARM ID\" is being passed in instead of the \"Incident ARM Id\" that is required.  The correct value for the parameter will be set and the playbook saved.

Rerunning a playbook

Rather than needing to wait until the incident is triggered again, you can rerun the playbook using old data.   Go back to the \"Overview\" page and look at the \"Runs history\" tab, which is selected by default.  Select the latest run and click its identifier to go into its run history.  At the top of the page will be a \"Resubmit\" button which will rerun the playbook using the same information that was used to run this instance.

When that button is clicked, the Notifications button at the top of the page will show that the playbook has been successfully resubmitted.   Wait a little while and then refresh the run history shown on the left side of the page.  If everything went OK, there will be a new successful entry added.   Click on that to see the successful run.

The image below shows this playbook with a successful run.

Debugging Conditional steps

The debugging steps listed above have limited usefulness when debugging a \"Condition\" step.   If you click on the step, it will only show you the result of the comparison, not what the comparison is or what the parameter values were.  The image below shows the playbook modified to use a condition step to only add the task to the incident if the alert's status is \"new\".  Most of the time, all the alerts inside an incident will have the same status; this was just used as an example.

Note that to access the alert's status, use the formula

items('For_each')['properties']['status']

There is also a JSON translation action that could be used, but that it outside the scope of this blog.

After rerunning this playbook, it shows that the playbook successfully ran, but there are no new tasks added to the incident.  By looking at the last run, and clicking on the \"Condition\" step, only the result's value is displayed, as shown below.

While this is useful to show what the conditional value was determined to be, it is not useful in that there is no way to determine how the result was computed.   By editing the workflow, the formula used in the condition can be determined (see the first image in this section), but not the value of the parameters used. 

To do that, the JSON in the \"For each\" step could be looked at to determine what the value is.  However, that would require knowing the structure of the JSON.  If this case, knowing the \"status\" field is under the \"properties\" field and then determine which instance of the alert is being viewed.

In this case, it may be easier to create a new variable and set its value to same parameter that the condition step is using as shown below.  Use the same formula for finding the alert's status when setting the variable as was used in the condition step shown above. 

Now, when looking at the run history, clicking on the \"Set variable\" step will show the value that is being used in the condition step as shown below.

Looking at this, it was determined that the parameter's value is \"New\" (with a capital \"N\") but it is being compared to \"new\" (with a lowercase \"N\").   Changing the text the parameter is being compared to in the condition step resolved this issue.

Trigger History

Sometimes Logic Apps do not get the correct data to properly execute the workflow.   This will not typically be the case with playbooks as all the data is being passed in from Microsoft Sentinel to the Logic App.  However, if you want to see all the data being passed into the playbook, select the \"Trigger history\" tab from the \"Overview\" page and then select one of the entries.   This will open a page like the one shown below.

Select the \"Outputs link\" link to see the JSON representation of all the data being passed into the playbook.  At times this can be faster than trying to see this by looking at the trigger in the actual workflow.

Summary

This blog post walked through one way to help debug a playbook.  There are other ways, including setting a variable to the different values, to help debug.

This article assumes knowledge of how to create Microsoft Sentinel Analytic Rules, Automation Rules, and playbooks.

Create the playbook

You will want to create the playbook first, since one will need to be created before it can be used in the Automation rule.   In the example shown below, the playbook will iterate through all the alerts in the incident and generate a task based on information in that alert.

Keep in mind that you will need to grant the Managed Identity access to the Microsoft Sentinel instance so that it can add the task to the incident.

Create sample Analytic rule

When you are creating a new playbook, you will need to test it.  Rather than waiting for the perfect incident to be created, you can create a dummy Analytic rule that utilizes the KQL \"datatable\" operator to generate the information you need to test your playbook.

An example of this is shown below.  This code sample was taken from the \"datatable\" operator's documentation located at datatable operator - Azure Data Explorer & Real-Time Analytics | Microsoft Learn

To trigger the loop in the playbook, set the \"Event Grouping\" to \"Trigger an alert for each event\" and enable \"Alert grouping\" and set it to \"Grouping all alerts trigger by this rule into a single incident\"

Create the Automation Rule

Create a new automation rule that triggers off the analytic rule that was created above.  This will make sure no other incidents will trigger the automation rule and playbook accidently while testing.   Once the playbook has been debugged, this automation rule can be modified to trigger off any other Analytic rules or all of them.

Set the \"Actions\" to run the playbook that was created above and save it.

Debugging the playbook

Once your analytic rule has triggered, you can disable it so that it does not create an excessive number of incidents.  This blog will show how to rerun a playbook with old data.

In the Microsoft Sentinel Automation Rules, select the \"Active playbooks\" tab and select your playbook.  You will see a page that look likes the image below showing how many runs have been performed and the status.   Note that in the image below, this playbook was run and had errors.   If it was successful, the \"status\" field will be set to \"Succeeded\"

Click on the identifier link to be brought into the playbook's history while will show the error. 

The full functionality of playbooks is outside the scope of this blog post, but note that by clicking the arrows in the loop, you can progress to the next error or the next value in the loop.  This is very useful if only certain entries in the loop are causing issues.

In this case, the error shown is \"BadRequest\".  This is a fairly generic error message but usually means that one of the parameters being set is wrong.  If you click on the step, a new pane will open showing all the information about the error including the inputs to the step and the output.  Use the \"INPUTS\" section to verify what parameters are being used and the \"OUTPUTS\" section to see what has been returned. 

Looking at the \"INPUTS\", shown below, doesn't provide too much useful information in this case.  It appears that an ARM Id is being passed in as \"incidentArmId\" and the \"taskTitle\" and \"taskDescription\" both have values.

Looking at the \"OUTPUTS\" section, shown below, provides a bit more useful information.  The \"innerError\" message states that \"Incident Arm id is invalid\" which provides a good place to start looking.   

To edit the playbook, close this run history page, and then in the \"Overview\" page, click the \"Edit\" button.

Click on the \"Add task to incident\" step to see what values are used.   This will open the parameters pane to see what parameters are being used as shown below

Notice that the \"Alert ARM ID\" is being passed in instead of the \"Incident ARM Id\" that is required.  The correct value for the parameter will be set and the playbook saved.

Rerunning a playbook

Rather than needing to wait until the incident is triggered again, you can rerun the playbook using old data.   Go back to the \"Overview\" page and look at the \"Runs history\" tab, which is selected by default.  Select the latest run and click its identifier to go into its run history.  At the top of the page will be a \"Resubmit\" button which will rerun the playbook using the same information that was used to run this instance.

When that button is clicked, the Notifications button at the top of the page will show that the playbook has been successfully resubmitted.   Wait a little while and then refresh the run history shown on the left side of the page.  If everything went OK, there will be a new successful entry added.   Click on that to see the successful run.

The image below shows this playbook with a successful run.

Debugging Conditional steps

The debugging steps listed above have limited usefulness when debugging a \"Condition\" step.   If you click on the step, it will only show you the result of the comparison, not what the comparison is or what the parameter values were.  The image below shows the playbook modified to use a condition step to only add the task to the incident if the alert's status is \"new\".  Most of the time, all the alerts inside an incident will have the same status; this was just used as an example.

Note that to access the alert's status, use the formula

There is also a JSON translation action that could be used, but that it outside the scope of this blog.

After rerunning this playbook, it shows that the playbook successfully ran, but there are no new tasks added to the incident.  By looking at the last run, and clicking on the \"Condition\" step, only the result's value is displayed, as shown below.

While this is useful to show what the conditional value was determined to be, it is not useful in that there is no way to determine how the result was computed.   By editing the workflow, the formula used in the condition can be determined (see the first image in this section), but not the value of the parameters used. 

To do that, the JSON in the \"For each\" step could be looked at to determine what the value is.  However, that would require knowing the structure of the JSON.  If this case, knowing the \"status\" field is under the \"properties\" field and then determine which instance of the alert is being viewed.

In this case, it may be easier to create a new variable and set its value to same parameter that the condition step is using as shown below.  Use the same formula for finding the alert's status when setting the variable as was used in the condition step shown above. 

Now, when looking at the run history, clicking on the \"Set variable\" step will show the value that is being used in the condition step as shown below.

Looking at this, it was determined that the parameter's value is \"New\" (with a capital \"N\") but it is being compared to \"new\" (with a lowercase \"N\").   Changing the text the parameter is being compared to in the condition step resolved this issue.

Trigger History

Sometimes Logic Apps do not get the correct data to properly execute the workflow.   This will not typically be the case with playbooks as all the data is being passed in from Microsoft Sentinel to the Logic App.  However, if you want to see all the data being passed into the playbook, select the \"Trigger history\" tab from the \"Overview\" page and then select one of the entries.   This will open a page like the one shown below.

Select the \"Outputs link\" link to see the JSON representation of all the data being passed into the playbook.  At times this can be faster than trying to see this by looking at the trigger in the actual workflow.

Summary

This blog post walked through one way to help debug a playbook.  There are other ways, including setting a variable to the different values, to help debug.

Microsoft Sentinel playbooks use Azure Logic Apps behind the scenes to perform workflow automation.  While they are powerful, they can be hard to debug when things go wrong.  This blog post will show some steps you can take to debug your playbooks.

Internetguy441 For your first comment, while that would be very useful, it would be rather difficult to do considering how different each incident can be from each other.  That is why I suggested creating a dummy Analytic Rule that will create an incident that contains exactly what you need for testing.  When you kick off the playbook from Sentinel, that initial step will have the parameters coming from the incident. You can look at the 'inputs' for that step to see the data being passed in.

Not sure I can help much with your statement regarding variables and loops.  I know I have had many variables when doing testing before, both inside and outside of loops.   The only suggestions I would make are to: 

1) Make your variable names meaningful

2) Initialize all the variables close to each other (makes it easier to find each one if you need to)

3) Set the value of the variable as close to the step you are looking at as possible to ensure all the dynamic values are set correctly.

Keep in mind that the same variable can be used in each iteration of the loop and will contain the value for that specific iteration.  So, if the loop loops 3 times, you do not need a separate variable for loop 1, loop 2, and loop 3.  

Hope this helps.

Thanks for the runthrough! 🙂 always usefull.
I have to add, for me who is learning to create some basic playbooks - it would be superhelpfull if it was possible to chose some modules (like the \"microsoft incident\" and be able to trigger it from the builder (ie: press \"run/play-button\") just to see response for that exact step). In some cases i could then copy the JSON response and \"generate template\" through the gui. 🙂

Also setting variables are still a mystery to me, im always a little confused on where to set them in my loop so i end up with 5-10 variables before doing anything.