Running workflows in Project Online involves plenty of moving parts and many configuration options that might affect how workflows flow – or maybe stop flowing. In this blog I aim to try and demystify some of the processes involved and help you understand why workflows may get suspended – and quite often it isn’t anything to do with any failure in our service. That’s not to say I am blaming customers – as we don’t make it at all easy to understand what happened – or why. We are working on that part – but for now…
If you are familiar with workflows and want to cut to the chase – scroll down a yard or so (just past the Excel screenshot) to get to where I start talking about the errors.
First lets cover where you can see status on workflows. From the main project workflow page itself, you can see the phase, stage and any workflow stage status that has been set through your workflow. In this case I’m not showing any specific stage status – but it is a good idea to put something here to let your users know what is going on – and maybe what it might be waiting for. Expanding the All Workflow Stages section as I have done here gives a broader view of where you are. Assuming you have the right permission level – you will see an option at the bottom right corner to see Additional Workflow Data:
Clicking the Additional Workflow Data option takes you to the next screen – and you will see any status information written to the workflow history list – and the internal status. If you are troubleshooting you might see the Internal Status show as suspended – and in my case this workflow would finally end up getting suspended after finishing the retries – as we can see the status under the information icon next to ‘Started’ in the subsequent screenshot.
You can also get to workflow status from Site Settings, Site Workflows and Show all workflows.
Unfortunately the page of workflows it shows are just displayed by workflow and have no direct project context – but you will be able to see which ones are suspended:
Another way you can get at workflow data is via odata – using _api/ProjectData/ProjectWorkflowStageDataSet added to your PWA Url. Here is a an Excel sheet I’ve created populated from that feed on my server, and I filtered the StageStatus down to just see errors and waiting – and some of the waiting ones would fail eventually – but I’m impatient…:
So that’s some background on where you can find status – so lets start with some common errors you might see – you may see these where a workflow goes into a suspended state – or you might see it in the interim when it is still going through a number of retries as above – but you see something like this if you view the workflow status. All of these would end up suspended after all the retires – unless the condition was rectified before the retry cycle completed:
The underlying cause of each of these errors was a permission problem with the account that was running the workflow. I can’t guarantee which error you might see in every scenario – as it is really a combination of the specific permission issue and the action the workflow is attempting. But which user account has the issue and is having their access denied?
When a project is created the workflow is kicked off using a token from the logged in user – and it continues to run in the context of that user until it completes – or for some reason fails. It still keeps this same user context even if the project owner changes! Things that can happen to induce a failure are lack of the right permissions to start with – and the last example fits this scenario – and it was the PDP that was triggering the ‘403’. This is fairly easy as it is usually obvious who is running the workflow if it fails right away. But what if the user who started the workflow leaves the company and becomes inactive? This can give rise to the Forbidden: Access Denied, and InternalServerError: GeneralSecurityAccessDenied messages. Restarting the workflows will resolve this issue. Another example I have seen is when the account that is running the workflow loses some permissions – maybe they changed from Project Manager to Team Member – or perhaps the category they were in was updated with more limited access (permissions or even projects!). The same two messages will appear, depending on the actions happening in the workflow.
How about this one?
Here we can see a little more information than the others. The workflow is reading some DateTime field and having a problem. We can see a propertyid referenced - '21af92c1-a389-e711-80c8-00155de44f0a' – and generally these will be referencing custom fields – usually at the project level. A quick way I tend to use for finding which custom field has that GUID is to use View Source on the Custom Field and Lookup Table page and search for the GUID. Here we can see I tracked it down to ProjDate:
In my workflow I was reading that date and setting a variable – but as the date hadn’t been set a NullReferenceException was thrown – which led to the InternalServerError that you see in the workflow. In this case ensuring that the date will have a value before reading it will avoid the problem.
The examples above are in most respects in the customers control – although if you weren’t aware that the user who started the workflow was still running it this probably wasn’t something you’d have considered. But we are looking to see if we can improve the experience here – so watch this space. We are also seeing if we can better deliver the true cause of the problem to ensure the errors are more actionable – and that you can tell if it is something you can resolve by just restarting or if you need to log a call as it is something broken on our end. An example of this you may have seen recently was this error:
RequestorId: 2d852b8a-41ad-5d86-0000-000000000000. Details: System.InvalidOperationException: Incomplete closure detected while loading subroutines for workflow 66149d57-9e52-4a69-970d-2324e6132ec1 in scope .
And explaining how the fix for this rolls out may help understand why you may sometimes continue to see things that we say we have fixed. The workflow team addressed this quite a while back – and they have some automated processes that also fix up the running (in most cases waiting or sleeping is a better description) workflow that may have the old copy of this code. To complicate this issue the automated process had some problems – so customers continued to see the problem way too long after it was fixed. We should be getting up to date with this fix – and again, a restart would have achieved the same result – getting the latest version of the code.
Hopefully this will help understand some of the mystery of suspended workflows and enable you to pick out which might need a case to resolve. I’ll be posting again when we have some news on better actionable errors and workflow handling.
For the search engines here is the text from the error screenshots above:
Retrying last request. Next attempt scheduled in less than one minute. Details of last request: HTTP InternalServerError to https://brismithpjo.sharepoint.com/sites/pwa/_api/ProjectServer/WorkflowActivities/ReadDateTimePrope...') Correlation Id: a7da1060-9fbd-8fe0-82e6-678a0b0a2b6e Instance Id: 99d1d76b-daf2-45b2-9489-c9abdb7cce69
Retrying last request. Next attempt scheduled after 8/25/2017 4:11 PM. Details of last request: HTTP InternalServerError to https://brismithpjo.sharepoint.com/sites/pwa/_api/ProjectServer/WorkflowActivities/ReadyToLeaveProje...') Correlation Id: a7da1060-9fbd-8fe0-a513-0efff7b5357b Instance Id: fc8f01bd-cb49-4cdc-9ed1-f53dff2d640f
ProjectServerError(s) LastError=GeneralSecurityAccessDenied Instructions: Pass this into PSClientError constructor to access all error information
Retrying last request. Next attempt scheduled after 8/25/2017 6:10 PM. Details of last request: HTTP Forbidden to https://brismithpjo.sharepoint.com/sites/pwa/_api/ProjectServer/WorkflowActivities/ReadyToLeaveProje...') Correlation Id: a7da1060-9fbd-8fe0-8554-9c3f3931dc99 Instance Id: 7620f076-f706-4692-80d3-6d02a75f9fff
Access denied. You do not have permission to perform this action or access this resource.
RequestorId: 9e975a9a-2037-bbfa-0000-000000000000. Details: An unhandled exception occurred during the execution of the workflow instance. Exception details: System.ApplicationException: HTTP 403 {"Transfer-Encoding":["chunked"],"X-SharePointHealthScore":["0"],"X-MSDAVEXT_Error":["917656; Access+denied.+Before+opening+files+in+this+location%2c+you+must+first+browse+to+the+web+site+and+select+the+option+to+login+automatically."],"X-SP-SERVERSTATE":["ReadOnly=0"],"DATASERVICEVERSION":["3.0"],"SPClientServiceRequestDuration":["74"],"SPRequestDuration":["142"],"SPRequestGuid":["9e975a9a-2037-bbfa-95aa-aa690c69f8a2"],"request-id":["9e975a9a-2037-bbfa-95aa-aa690c69f8a2"],"Strict-Transport-Security":["max-age=31536000"],"X-FRAME-OPTIONS":["SAMEORIGIN"],"MicrosoftSharePointTeamServices":["16.0.0.6809"],"X-Content-Type-Options":["nosniff"],"X-MS-InvokeApp":["1; RequireReadOnly"],"Cache-Control":["max-age=0, private"],"Date":["Fri, 18 Aug 2017 09:33:41 GMT"],"P3P":["CP=\"ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI\""],"Server":["Microsoft-IIS\/8.5"],"X-AspNet-Version":["4.0.30319"],"X-Powered-By":["ASP.NET"]} at System.Activities.Statements.Throw.Execute(CodeActivityContext context) at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation) Exception from activity Throw If Sequence Sequence TryCatch Sequence Microsoft.SharePoint.WorkflowServices.Activities.RetryForDurationPolicy HttpPost Sequence Microsoft.Office.Project.Server.WorkflowActivities.EnterProjectStage Sequence Flowchart Sequence BSTest.WorkflowXaml_f9e81a98_8151_4254_a524_eab9f8e99079