Thanks to Marc for the original research and posting this item on the French blog - http://blogs.technet.com/b/frenchpjblog/archive/2012/12/24/3542467.aspx and to Jorge for the Spanish version at http://blogs.technet.com/b/elfarodeprojectserver/archive/2012/12/26/project-server-2010-se-para... .
For those whose English is better than their French or Spanish – read on. This is something we have seen quite a few calls on, and the initial symptom is that Project Server 2010 queue jobs were not getting processed, they were just ‘waiting to be processed’. Nothing appeared to be blocking these queue jobs and no ‘processing’ jobs were seen. One closer examination, even though the Project Application Service was started within Central Administration – if you actually looked in the Services applet in the operating system you would see that the Services "Microsoft Project Server Event Service 2010" and "Microsoft Project Server Queue Service 2010" were not only stopped, but disabled.
And even if you enabled and restarted the service they would get disabled again – and the queue would stop processing.
Marc’s research showed that the disabling of the service occurred when the Hourly Health Analysis job was executed by the SharePoint TImer service;
specifically the rule that checks for services that have started or stopped unexpectedly;
To validate if this is indeed the issue causing your queue problems (as there could be other causes) you can just select the rule and ‘Run now’. You should then see the services stopped and disabled and in the ULS logs the following rows can be found:
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
Health |
ag78 |
Verbose |
Checking the Microsoft Project Server Queuing Service windows service instance. |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
General |
0000 |
Verbose |
Entered SPAdvApi32.IsServiceRunning(ProjectQueueService14) |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
Health |
ag7d |
Verbose |
The service is not disabled, but should be. |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
Health |
8fs1 |
Verbose |
Finished invoking the Check() method. The rule Failed |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
Health |
8fs4 |
Medium |
Automatic repair is being attempted. |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
General |
0000 |
Verbose |
Entered SPAdvApi32.IsServiceRunning(SPAdminV4) |
|
OWSTIMER.EXE (0x1A60) |
0x0C48 |
SharePoint Foundation |
General |
0000 |
Verbose |
Entered SPAdvApi32.StopService(ProjectQueueService14) |
|
Microsoft.Office.Project.Server (0x1A08) |
0x22B0 |
Project Server |
General |
8zdx |
High |
[SERVICE] ProjectQueueService14: shutting down |
The rule is somehow detecting that the service should be disabled and so is making the change. Again, Marc did some great detective work here and discovered that even though the Service Application was ‘Started’ as per the screen shot above, the following PowerShell commands would return ‘Disabled’
((Get-SPFarm).)Services| where {$ _.}Name - match "ProjectQueueService14"}) .instances
((Get-SPFarm).)Services| where {$ _.}Name - match "ProjectEventService14"}) .instances
This inconsistency in the state of these services was leading to them being disabled by the Health check.
We still have no good explanation why this situation occurs, but there is a way to stop it happening. Again we use PowerShell and we need to run the following command on all the servers on which the services are installed:
Start-SPServiceInstance - Identity < Id >
where Id is the Id returned by the earlier Get-SPFarm command for the specific services, Sometimes we have seen that this will still not resolve the issue immediately and you may need to clean the configuration cache and then stop and start the service instance:
Stop-SPServiceInstance - Identity < Id >
Start-SPServiceInstance - Identity < Id >
As mentioned earlier – there can be other reasons for a slow or stopped queue – see http://blogs.msdn.com/b/brismith/archive/2012/09/19/when-your-project-server-queue-slows-down.a... for some examples, but glad that Marc managed to clear this one up. Thanks again Marc, and to Jorge for reposting in Spanish.