In a peculiar case, we noticed that after an in-place upgrade from Lync Server 2013 to Skype for Business Server 2015, Response Group Workflows stopped working. The initial troubleshooting involved verifying (i) that the Response Group Service is starting, (ii) the work-flow is defined in Skype for Business Server 2015 after the upgrade, (iii) the AD Object for the workflow exists.
To troubleshot further, as a simple step, we deleted and re-created the workflow, and could reproduce the error. From the logs on client, we could see 500 Internal Server Error , so I wanted to first make sure the permissions on AD are set correctly based on Issue: Calls to certain Response groups fail with an Error "500 Internal Server Error" . Once I could verify that the issue was indeed not to do with AD permissions, I started to investigate further.
Looking through the built-in scenarios, a scenario called RGS exists, but I wasn’t sure, if the components would suffice.
Description | Centralized Logging Tool Scenario | Logging Tool Components |
Response Group Service | RGS | RgsClientsLib |
I decided to create a custom scenario, call it RGSCustom and perform logging. I used the below commands to create my custom scenario
$SIPStack = New-CsClsProvider -Name "SIPStack" -Type "WPP" -Level "Debug" -Flags "All"
$S4 = New-CsClsProvider -Name "S4" -Type "WPP" -Level "Debug" -Flags "All"
$RGSClientLib = New-CsClsProvider -Name "RGSClientLib" -Type "WPP" -Level "Debug" -Flags "All"
$RgsCommonLibrary = New-CsClsProvider -Name "RgsCommonLibrary" -Type "WPP" -Level "Debug" -Flags "All"
$RgsDatastores = New-CsClsProvider -Name "RgsDatastores" -Type "WPP" -Level "Debug" -Flags "All"
$RgsDeploymentApi = New-CsClsProvider -Name "RgsDeploymentApi" -Type "WPP" -Level "Debug" -Flags "ALL"
$RgsDeploymentLibrary = New-CsClsProvider -Name "RgsDeploymentLibrary" -Type "WPP" -Level "Debug" -Flags "ALL"
$RgsDiagnostics = New-CsClsProvider -Name "RgsDiagnostics" -Type "WPP" -Level "Debug" -Flags "All"
$RgsHostingFramework = New-CsClsProvider -Name "RgsHostingFramework" -Type "WPP" -Level "Debug" -Flags "ALL"
$RgsMatchMakingService = New-CsClsProvider -Name "RgsMatchMakingService" -Type "WPP" -Level "Debug" -Flags "All"
$RgsDBSyncAgent = New-CsClsProvider -Name "RgsDBSyncAgent" -Type "WPP" -Level "Debug" -Flags "ALL"
New-CsClsScenario -Identity "Global/RGSCustom" -Provider @{Add=$SIPStack,$S4,$RGSClientLib,$RgsCommonLibrary,$RgsDatastores,$RgsDeploymentApi,$RgsDeploymentLibrary, $RgsDiagnostics, $RgsHostingFramework, $RgsMatchMakingService, $RgsDBSyncAgent}
Next it was time to start logging and while we were logging to restart the Response Group Service, so that I would have any failure encountered during service state captured
Start-CsClsLogging -Scenario "RGSCustom" -Pools SFBSTD01.contoso.com
Stop-CsClsLogging -Scenario "RGSCustom"
Search-CsClsLogging -Pools SFBSTD01.contoso.con -OutputFilePath "C:\Windows\Temp\RGSCustom.txt" -StartTime (get-date).AddMinutes(-30)
As I was investigating the CLS logs, I noticed the a peculiar error “ Message: CALLCONTROL: Call declined because CallControl is not started ”
TL_INFO(TF_COMPONENT) [SFBSTD01\SFBSTD01] 208D8.4700::05/03/2016-19:46:01.484.0000A42D (RgsCommonLibrary,RgsLogMessage.ReportMessageInternal:rgslogmessage.cs(452))
_rgs_message_begin_
Direction: Incoming
From: sip:sritodi@contoso.com
To: sip:sampleworkflow@contoso.com
Message: CALLCONTROL: Call declined because CallControl is not started
WorkflowId: c338e8ba142042b9b30023269d29daa0
_rgs_message_end_
I decided to look through the event logs when the service was starting and noticed EVENT ID 31067
This sounded a little strange as I was aware that every Lync Server 2013 pool has 2 Application Contacts, one RGS Presence Watcher and another RGS Announcement Service. This event was speaking about 3 RGS Presence Watcher Contact objects.
I decided to query the number of RGS Presence Watcher contact objects
Get-CsApplicationEndpoint | Where-Object DisplayName -eq "RGS Presence Watcher" | Ft Identity, DisplayName, RegistrarPool
This could also be accomplished by running
Get-ADObject -Filter 'msRTCSIP-OwnerUrn -eq "urn:application:RGS"' -SearchBase 'CN=Configuration,DC=contoso,DC=Com' -Properties msRTCSIP-PrimaryUserAddress, displayName | ft msRTCSIP-PrimaryUserAddress, displayName –AutoSize
I quickly realized that for the pools that were upgraded from Lync Server 2013 to Skype for Business Server 2015 and where calls to response group were failing, only had a single RGS Presence Watcher contact object
Solution: We decided to republish the topology by running
Enable-CsTopology -Verbose
Once topology was republished, we could see 3 RGS Presence Watcher contact objects. A service restart to the RGS service, and calls started to work as expected.