Starting with Skype for Business Cloud Connector (Cloud Connector) 1.4.1, we introduced an automated update process: Cloud Connector automatically update based on the update schedule that administrators have configured for their Cloud Connector Hybrid PSTN Sites. This article goes into the details of this automatic update.
Note: Cloud Connector should be viewed as a part of Skype for Business Service. We are constantly improving the service and make changes in the Service and in Cloud Connector. If you do not update the Cloud Connector to the latest release, you might end up in the situation when your Cloud Connector will not work properly. Imagine the situation, Microsoft updated the service, added changes in Cloud Connector code to make it work with the new version of the service, but if you don’t update the Cloud Connector, your telephony will not work. Because of these, Microsoft supports only the latest version of Cloud Connector software. Also, to accommodate Update Window, we support the N-1 version for 60 days after releasing a new version.
Auto Update Requirements
- Outbound internet access to install, manage, and update Cloud Connector on Host Appliance.
- Outbound internet access on all Cloud Connector VM’s to download Windows updates, or, access to WSUS server as defined in Cloud Connector configuration file.
- Skype for Business Online PowerShell Module installed on Host Appliance.
- CCE Management Service is running on Host Appliance.
- Group Policy to prevent forcefully unloading user registry at log off (required for 1.4.1).
- Skype for Business Tenant Admin account.
Initial Setup
When the first Cloud Connector appliance is registered, a PSTN site is configured in the Office 365 Skype for Business Tenant, with the name of the Site as defined in the Cloud Connector configuration file Auto update is enabled, and default time windows are configured for both the software (bits) and operating system (windows) with corresponding Tenant Update Time Windows that are configured in the tenant as part of this registration process. See the image below.
Confirm or Modify the Update Schedule for Hybrid PSTN Site(s)
To confirm the schedule for updating Cloud Connector software, administrators can view the Hybrid PSTN site update schedule in Skype for Business Admin Center in the on premises PSTN tab of the Voice section. Note that updates will run based on the local host time of the Cloud Connector appliance.
Please be sure to confirm that auto update is enabled and that a bits update time window is set to correspond to the maintenance window that you want the updates to run. The time that updates will occur is based on the local time of the Cloud Connector host appliance. For example, if the update time window is set for 11PM and the Cloud Connector appliance is in Amsterdam, the updates will occur at 11PM CET (UTC+1).
For details on how to configure update time windows, please refer to Modify the configuration of an existing Cloud Connector deployment.
Cloud Connector Update Process
The CCE Management service uses cached update time window information stored in the root of the CCE Site Directory\Tenant_<EdgeFQDN> file when checking if updates run. This file is updated from O365 tenant every 30 minutes. Therefore, if you modify the schedule online, it can take up to 30 minutes for the change to be implemented on the host appliance.
Overall Auto Update Process
- The auto update process will run on the schedule set by Tenant Administrator based on local host appliance clock
- Update detection will continue to run for the duration specified in the update time window.
- If an update is detected, the update process is invoked.
-
The appliance will be put in maintenance mode and only one appliance per site can enter maintenance mode. The maintenance mode lock is written to the root of the CCE Appliance Directory\CceSevicePersistent file
{"AutoMaintenanceStatus":#,"IsInManualMaintenance":false} 0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode -
Update tasks will run.
-
Once updates are completed and all services are confirmed running, the Appliance will be taken out of maintenance mode.
-
Repeat steps for the next appliance in site.
Monitor Update Process
The Cloud Connector management service will log events to the Windows Application log with a source of CCEManagementService and detailed information will be written to "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService\CceManagementService.log".
Note: the CCEManagementService.log can grow quite large, so you might want to stop the CCE Management Service, and rename this log periodically. There are plans to modify logging in the future to prevent log growth. If the log file size becomes too large to open in a text editor, you must use a text file splitter to break into smaller segments.
You can also see the status of the appliance by running Get-CsPSTNHybridAppliance in Remote PowerShell or by viewing in on premises PSTN tab in the Voice section of the Skype for Business Admin Center.
Bits Update Process
During this process, the running version remains in service, and an interim switch is used to connect to the new VM’s. Once the new version installation is complete and services are confirmed to be running, the old version is drained stopped and the network connections are switched to the new version.
- Bits update is detected based on the scheduled time window.
- Bits update task is triggered.
- The Cloud Connector download site is queried and if a new build is detected, then the update will occur.
- The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and DeploymentStatus of Upgrading, with the new version and the start time that the update began.
- Cloud Connector bits are downloaded.
- The CCE management service is stopped.
- The Skype for Business Online Cloud Connector edition software is updated which requires uninstalling the old version and installing the new version.
- New virtual machines are built from the existing VHDX file. If the VHDX is detected to be older than 90 days, the Install Instance script will log the following warning:
- SFBServer.vhdx was generated more than 90 days before. Use Convert-CcIsoToVhdx to generate it again and apply windows updates.
-
Note: It is recommended that a new VHDX be built periodically to reduce the amount of time to perform Windows updates for new and updated Cloud Connector machines. It’s not supported to update the VHDX with Windows update and re-run Sysprep as there are a limited number of times that Sysprep can run on a computer.
- Once the deployment of the new Cloud Connector is completed and services confirmed running, the switch to the new version will occur as follows:
- Change virtual network connections to new Cloud Connector virtual machines
- Shut down the N+1 version
- Remove N+2 version and delete the virtual disks.
- The appliance will be taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of running, Version number of new build, and DeploymentStatus of Upgraded.
- Detailed logs for the download, upgrade of the Cloud Connector software, new version installation and switch to new build will be written to the Logs folder located in the root of the Appliance directory.
Windows Update Process
Windows update process is performed on the active running version. Therefore, when a windows update is detected, the appliance is drained stopped and put in maintenance mode.
- OS update is detected during scheduled time window.
- OS update task is triggered.
- The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and OsUpdateStatus of Upgrading, with start time.
- The RTCSRV service on Edge and the RTCSRV and RTCMEDSRV services on Mediation server are drained stopped.
- OS update PowerShell script is copied to the root of the System drive on all CCE VM’s.
- Local windows update service is triggered to check for updates either against Windows Updates Internet service, or the local WSUS server defined in Cloud Connector configuration file.
- Updates are installed and a check for virtual machine restart is run.
If a restart is required, all Cloud Connector virtual machines are restarted, then a second check for restart is run. - Once updates are completed on all virtual machines, the updates are run on the host appliance and its restarted.
- Once the host has been restarted and no additional restarts are confirmed, the appliance is taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of Running, and OSUpdateStatus of Updated.
CCE Management Service Logging Level: If you need more diagnostic logging, you can modify the logging level to verbose for the following two settings in the Microsoft.Rtc.CCE.ManagementService.exe.config located in the "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService folder. (This will cause rapid log growth):
<add name="serviceSwitch" value="Information"/> <add name="powershellSwitch" value="Warning"/>
If updates are not running because another maintenance task is detected, check the status of the CCE Appliance Directory\CceSevicePersistent file to determine what task is running.
{"AutoMaintenanceStatus":#,"IsInManualMaintenance":false}
0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode
Bits update failed to switch version is logged by the CCE Management Service with following error:
CceService Error: 20003: Bits update failed to switch version. Appliance running status: Running,
error detail: Failed to drain services with exception: [192.168.213.4] Connecting to remote server192.168.213.4 failed with the following error message: Access is denied
Check the networking status on the virtual machines and be sure there are no duplicate IP’s configured.