First published on TECHNET on Apr 20, 2017
*** Update 4/27 - further testing and this does not affect Project Server 2016 - and therefore also unlikely to affect Project Online (although I haven't tested) ***
This blog post comes out of a recent support incident where a customer had configured a server side event handler on the Statusing Approvals Updating event – and had configured a remote event receiver. This type of event handler uses a web based endpoint – either on the same server as Project Server or another server – or even in the cloud – rather than the more commonly seen event handlers (at least for on-premises installations) that are based on locally installed assemblies. This issue can affect Project Server 2013, as well as Project Server 2016 – and at this time we are not fixing the issue in any upcoming patch but are sharing the information here (and will update other documentation too). It does not affect the local assembly based event handlers.
So enough of that – what’s the issue? In this case if a large number of approvals were listed in Approval Center (probably 40+ would be enough to trigger the problem) then when an item or items are approved the complete dataset passes across to the remote event receiver. The remote code does its job and then passes back the result (could be a cancel request or just a change of data – depending on the business logic). This is the point where the process was failing – and the error in this case was “An error occurred while processing one or more items”. It really depends on the event in question as the the precise error one might see when experiencing this problem – but the key aspect is having a large payload – so a lot of something going over the the remote event receiver and coming back.
The corresponding ULS errors are:
The problem was found to be that the proxy waiting for the response from the remote event receiver was using the default http binding and the maxReceivedMessageSize in that default is 65536. Once you have 20+ approvals then this size of message can be exceeded. There probably aren’t too many events in Project that would have messages that exceed this – but I’m sure approvals updating isn’t the only one that could exceed this limit.
The fix is easy and can be implemented on the server by editing the config file for the Microsoft Project Server Events service. The file is called Microsoft.Office.Project.Server.Eventing.exe.config and can be found (in a default installation) at C:\Program Files\Microsoft Office Servers\15.0\Bin for Project Server 2013 – or C:\Program Files\Microsoft Office Servers\16.0\Bin for Project Server 2016. Make a copy before you edit – and then just add the following XML snippet one line from the end - between the </runtime> and </configuration> tags.
Once the file is edited (on all servers) then you will need to restart the Microsoft Project Server Events Service 2013 service (Or the 2016 version if you are hitting this issue on 2016) on all servers where it is running and execute an IISReset to be sure the new value is picked up.
With this correction in place I can now approve my hundreds of pending approvals and they flow through my remote event receiver and back again perfectly!
It is unlikely that we’d make any changes to this file in a patch (and we are not currently planning to make this update in a future PU) – but potentially if we did you’d need to re-do the correction. I’ll be seeing where it makes sense to also add these details – such as TechNet or any Remote Event receiver documents.
*** Update 4/27 - further testing and this does not affect Project Server 2016 - and therefore also unlikely to affect Project Online (although I haven't tested) ***
This blog post comes out of a recent support incident where a customer had configured a server side event handler on the Statusing Approvals Updating event – and had configured a remote event receiver. This type of event handler uses a web based endpoint – either on the same server as Project Server or another server – or even in the cloud – rather than the more commonly seen event handlers (at least for on-premises installations) that are based on locally installed assemblies. This issue can affect Project Server 2013, as well as Project Server 2016 – and at this time we are not fixing the issue in any upcoming patch but are sharing the information here (and will update other documentation too). It does not affect the local assembly based event handlers.
So enough of that – what’s the issue? In this case if a large number of approvals were listed in Approval Center (probably 40+ would be enough to trigger the problem) then when an item or items are approved the complete dataset passes across to the remote event receiver. The remote code does its job and then passes back the result (could be a cancel request or just a change of data – depending on the business logic). This is the point where the process was failing – and the error in this case was “An error occurred while processing one or more items”. It really depends on the event in question as the the precise error one might see when experiencing this problem – but the key aspect is having a large payload – so a lot of something going over the the remote event receiver and coming back.
The corresponding ULS errors are:
04/20/2017 11:00:58.00 Microsoft.Office.Project.Server (0x8148) 0x9AC4 Project Server Server-Side Events ac35f Medium SEP: Calling endpointUrl http://localhost:34282/StatusEvents.svc failed. Error 'Exception has been thrown by the target of an invocation.' StackTrace ' at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Delegate.DynamicInvokeImpl(Object[] args) at Microsoft.Office.Project.Server.Events.ServerEventProxy.CallRemoteEventReceiver(DelegateInfo del, PSContextInfo contextInfo, String name, IPreEventArgs args) at Microsoft.Office.Project.Server.Events.ServerEventProxy.FirePreDelegate(PSContextInfo contextInfo, String name, IPreEventArgs args)'
04/20/2017 11:00:58.00 Microsoft.Office.Project.Server (0x8148) 0x9AC4 Project Server Server-Side Events 7941 Information Operation canceled by event handler: assembly: http://localhost:34282/StatusEvents.svc class: Statusing event: StatusingApprovalsUpdating handler: Microsoft.Office.Project.Server.Events.StatusingApprovalsUpdatingHandler, method: OnApprovalsUpdating
04/20/2017 11:00:58.03 w3wp.exe (0x6358) 0x6F0C Project Server Server-Side Events aisz6 Warning Standard Information:PSI Entry Point: Project User: i:0#.w|domain\user Correlation Id: b296e649-f325-e711-810d-001dd8b73698 PWA Site URL: http://<myserver>/Event SA Name: Project Server Service Application PSError: GeneralActionCanceledByEventHandler (22000) Action canceled by event handler. Reason: (null). 30ebe99d-aae4-e0fd-0000-09216c843577
The problem was found to be that the proxy waiting for the response from the remote event receiver was using the default http binding and the maxReceivedMessageSize in that default is 65536. Once you have 20+ approvals then this size of message can be exceeded. There probably aren’t too many events in Project that would have messages that exceed this – but I’m sure approvals updating isn’t the only one that could exceed this limit.
The fix is easy and can be implemented on the server by editing the config file for the Microsoft Project Server Events service. The file is called Microsoft.Office.Project.Server.Eventing.exe.config and can be found (in a default installation) at C:\Program Files\Microsoft Office Servers\15.0\Bin for Project Server 2013 – or C:\Program Files\Microsoft Office Servers\16.0\Bin for Project Server 2016. Make a copy before you edit – and then just add the following XML snippet one line from the end - between the </runtime> and </configuration> tags.
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="RemoteEventService" closeTimeout="00:05:00"
openTimeout="00:05:00" receiveTimeout="00:05:00" sendTimeout="00:05:00"
maxBufferPoolSize="524288" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
<readerQuotas maxDepth="32" maxStringContentLength="2147483647"
maxArrayLength="2147483646" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</basicHttpBinding>
</bindings>
</system.serviceModel>
Once the file is edited (on all servers) then you will need to restart the Microsoft Project Server Events Service 2013 service (Or the 2016 version if you are hitting this issue on 2016) on all servers where it is running and execute an IISReset to be sure the new value is picked up.
With this correction in place I can now approve my hundreds of pending approvals and they flow through my remote event receiver and back again perfectly!
It is unlikely that we’d make any changes to this file in a patch (and we are not currently planning to make this update in a future PU) – but potentially if we did you’d need to re-do the correction. I’ll be seeing where it makes sense to also add these details – such as TechNet or any Remote Event receiver documents.
Published Mar 06, 2019
Version 1.0
