Blog Post

Skype for Business Blog
7 MIN READ

Understanding Cloud Connector Edition Auto Update

Carolyn Blanding (MS TEAMS)'s avatar
Feb 23, 2017

Starting with Skype for Business Cloud Connector (Cloud Connector) 1.4.1, we introduced an automated update process: Cloud Connector automatically update based on the update schedule that administrators have configured for their Cloud Connector Hybrid PSTN Sites. This article goes into the details of this automatic update.

 

Note: Cloud Connector should be viewed as a part of Skype for Business Service. We are constantly improving the service and make changes in the Service and in Cloud Connector. If you do not update the Cloud Connector to the latest release, you might end up in the situation when your Cloud Connector will not work properly. Imagine the situation, Microsoft updated the service, added changes in Cloud Connector code to make it work with the new version of the service, but if you don’t update the Cloud Connector, your telephony will not work. Because of these, Microsoft supports only the latest version of Cloud Connector software. Also, to accommodate Update Window, we support the N-1 version for 60 days after releasing a new version.

 

Auto Update Requirements

  • Outbound internet access to install, manage, and update Cloud Connector on Host Appliance.
  • Outbound internet access on all Cloud Connector VM’s to download Windows updates, or, access to WSUS server as defined in Cloud Connector configuration file.
  • Skype for Business Online PowerShell Module installed on Host Appliance.
  • CCE Management Service is running on Host Appliance.
  • Group Policy to prevent forcefully unloading user registry at log off (required for 1.4.1).
  • Skype for Business Tenant Admin account.

Initial Setup

When the first Cloud Connector appliance is registered, a PSTN site is configured in the Office 365 Skype for Business Tenant, with the name of the Site as defined in the Cloud Connector configuration file Auto update is enabled, and default time windows are configured for both the software (bits) and operating system (windows) with corresponding Tenant Update Time Windows that are configured in the tenant as part of this registration process. See the image below.

 

Confirm or Modify the Update Schedule for Hybrid PSTN Site(s)

To confirm the schedule for updating Cloud Connector software, administrators can view the Hybrid PSTN site update schedule in Skype for Business Admin Center in the on premises PSTN tab of the Voice section. Note that updates will run based on the local host time of the Cloud Connector appliance.

Please be sure to confirm that auto update is enabled and that a bits update time window is set to correspond to the maintenance window that you want the updates to run. The time that updates will occur is based on the local time of the Cloud Connector host appliance. For example, if the update time window is set for 11PM and the Cloud Connector appliance is in Amsterdam, the updates will occur at 11PM CET (UTC+1).

For details on how to configure update time windows, please refer to Modify the configuration of an existing Cloud Connector deployment.

 

Cloud Connector Update Process

The CCE Management service uses cached update time window information stored in the root of the CCE Site Directory\Tenant_<EdgeFQDN> file when checking if updates run. This file is updated from O365 tenant every 30 minutes. Therefore, if you modify the schedule online, it can take up to 30 minutes for the change to be implemented on the host appliance.

 

Overall Auto Update Process

 

  1. The auto update process will run on the schedule set by Tenant Administrator based on local host appliance clock
  2. Update detection will continue to run for the duration specified in the update time window.
  3. If an update is detected, the update process is invoked.
  4. The appliance will be put in maintenance mode and only one appliance per site can enter maintenance mode. The maintenance mode lock is written to the root of the CCE Appliance Directory\CceSevicePersistent file

    {"AutoMaintenanceStatus":#,"IsInManualMaintenance":false}
    0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode
    
  5. Update tasks will run.

  6. Once updates are completed and all services are confirmed running, the Appliance will be taken out of maintenance mode.

  7. Repeat steps for the next appliance in site.

Monitor Update Process

The Cloud Connector management service will log events to the Windows Application log with a source of CCEManagementService and detailed information will be written to "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService\CceManagementService.log".

Note: the CCEManagementService.log can grow quite large, so you might want to stop the CCE Management Service, and rename this log periodically.  There are plans to modify logging in the future to prevent log growth. If the log file size becomes too large to open in a text editor, you must use a text file splitter to break into smaller segments.

You can also see the status of the appliance by running Get-CsPSTNHybridAppliance in Remote PowerShell or by viewing in on premises PSTN tab in the Voice section of the Skype for Business Admin Center.

 

Bits Update Process

During this process, the running version remains in service, and an interim switch is used to connect to the new VM’s. Once the new version installation is complete and services are confirmed to be running, the old version is drained stopped and the network connections are switched to the new version.

 

  1. Bits update is detected based on the scheduled time window.
  2. Bits update task is triggered.
  3. The Cloud Connector download site is queried and if a new build is detected, then the update will occur.
  4. The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and DeploymentStatus of Upgrading, with the new version and the start time that the update began.
  5. Cloud Connector bits are downloaded.
  6. The CCE management service is stopped.
  7. The Skype for Business Online Cloud Connector edition software is updated which requires uninstalling the old version and installing the new version.
  8. New virtual machines are built from the existing VHDX file. If the VHDX is detected to be older than 90 days, the Install Instance script will log the following warning:
    • SFBServer.vhdx was generated more than 90 days before. Use Convert-CcIsoToVhdx to generate it again and apply windows updates.
    • Note: It is recommended that a new VHDX be built periodically to reduce the amount of time to perform Windows updates for new and updated Cloud Connector machines. It’s not supported to update the VHDX with Windows update and re-run Sysprep as there are a limited number of times that Sysprep can run on a computer.

  9. Once the deployment of the new Cloud Connector is completed and services confirmed running, the switch to the new version will occur as follows:
    1. Change virtual network connections to new Cloud Connector virtual machines
    2. Shut down the N+1 version
    3. Remove N+2 version and delete the virtual disks.
  10. The appliance will be taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of running, Version number of new build, and DeploymentStatus of Upgraded.
  11. Detailed logs for the download, upgrade of the Cloud Connector software, new version installation and switch to new build will be written to the Logs folder located in the root of the Appliance directory.

Windows Update Process

Windows update process is performed on the active running version. Therefore, when a windows update is detected, the appliance is drained stopped and put in maintenance mode.

  1. OS update is detected during scheduled time window.
  2. OS update task is triggered.
  3. The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and OsUpdateStatus of Upgrading, with start time.
  4. The RTCSRV service on Edge and the RTCSRV and RTCMEDSRV services on Mediation server are drained stopped.
  5. OS update PowerShell script is copied to the root of the System drive on all CCE VM’s.
  6. Local windows update service is triggered to check for updates either against Windows Updates Internet service, or the local WSUS server defined in Cloud Connector configuration file.
  7. Updates are installed and a check for virtual machine restart is run.
    If a restart is required, all Cloud Connector virtual machines are restarted, then a second check for restart is run.
  8. Once updates are completed on all virtual machines, the updates are run on the host appliance and its restarted.
  9. Once the host has been restarted and no additional restarts are confirmed, the appliance is  taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of Running, and OSUpdateStatus of Updated.
Troubleshooting Auto Update

CCE Management Service Logging Level: If you need more diagnostic logging, you can modify the logging level to verbose for the following two settings in the Microsoft.Rtc.CCE.ManagementService.exe.config located in the "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService folder. (This will cause rapid log growth):

<add name="serviceSwitch" value="Information"/>
<add name="powershellSwitch" value="Warning"/>

 

If updates are not running because another maintenance task is detected, check the status of the CCE Appliance Directory\CceSevicePersistent file to determine what task is running.

{"AutoMaintenanceStatus":#,"IsInManualMaintenance":false}
0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode

Bits update failed to switch version is logged by the CCE Management Service with following error:

CceService Error: 20003: Bits update failed to switch version. Appliance running status: Running,
error detail: Failed to drain services with exception: [192.168.213.4] Connecting to remote server192.168.213.4 failed with the following error message: Access is denied

Check the networking status on the virtual machines and be sure there are no duplicate IP’s configured.

 

Updated Feb 23, 2017
Version 7.0
  • nkv nkv's avatar
    nkv nkv
    Copper Contributor

    Hi Carolyn,

     

    Thank you for this article describing the update/upgrade process, useful to know.

    Can you please add some update issues FAQ, mainly for multiple CCE on a site trying to upgrade to 1.4.2 through web proxy, I am stuck with both CCE displaying in their DeploymentStatus:

    Failed to download installation files with exception: Exception callling \"DownloadFile\" with \"2\" argument(s): \"Unable to connect to the remote server\"

     

    Exception calling "DownloadFile" with "2" argument(s): "Unable to connect to the remote server"

    At C:\Program Files\WindowsPowerShell\Modules\CloudConnector\Internal\MtDownloadCommon.ps1:160

    char:5

    +     $webclient.DownloadFile($url,$dest)

    +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

        + FullyQualifiedErrorId : WebException

     

    Don't know how to exit this RegisteredAndFailed & Maintenance mode for one CCE, the other still displaying Running mode but same error.

    Thanks.

  • Per posting in TechNet, did you resolve connectivity to Download site by running netsh winhttp set proxy bypass-list="192.168.213.1;192.168.213.2;192.168.213.3;192.168.213.4;192.168.213.5". Note that the bypass list is for default management switch IP's. If your cloud connector configuration was defined with different management switch subnet, replace those addresses with correct IP's.
  • nkv nkv's avatar
    nkv nkv
    Copper Contributor

    Thank you for this answer, I confirm the complete subnet on each CCE (192.168.213.*, and 192.168.214.*) has been set.

    I saw in the meantime the Technet has been updated to define proxy at computer level and not user level, through a registry or GPO. will try that and post result.

  • nkv nkv's avatar
    nkv nkv
    Copper Contributor

    Hi,

    to not leave a question unanswered, issue came from accounts' password desync it seems.

    After deleting CCE xml password files and re-register ccappliances to recreate these files with correct passwords, all went back to normal and upgrades were able to be performed.

    Many thanks for directions.

  • Douglas Jones's avatar
    Douglas Jones
    Copper Contributor
    Should Windows update be enabled on each VM when using the AUTO UPDATE feature or does one not have anything to do with the other? If they do not have anything to do with the other, should windows updates be enabled on each VM or no?
  • Trent's avatar
    Trent
    Iron Contributor

    Where do I determine what my CCE Site Directory path is?

     

    Cloud Connector Update Process
    
    The CCE Management service uses cached update time window information stored in the root of the CCE Site Directory\Tenant_<EdgeFQDN> file when checking if updates run. This file is updated from O365 tenant every 30 minutes. Therefore, if you modify the schedule online, it can take up to 30 minutes for the change to be implemented on the host appliance.

     

  • Hi Trent,

     

    You can determine the CCE Site Directory path by running Get-CcSiteDirectory from Administrative PowerShell on the CCE host appliance.