Multiple per device RDS CALS are issued the same device issue…

First published on TECHNET on May 08, 2015

Hello AskPerf! Ishu Sharma here again from Microsoft Performance team.  Today I will be discussing an issue where multiple per device Remote Desktop Services CALS are issued to the same device.
Before we dive into this topic, I would like to recall the below facts about RDS Per Device Licensing.

If an unlicensed client connects to a Remote Desktop Server for the first time, the Remote Desktop Licensing Server issues the client a temporary RDS Client Access License (CAL). After the user has logged into the session, the RDS server instructs the License Server to mark the issued temporary RDS CAL token as being validated. The next time the client connects, an attempt is made to upgrade the validated temporary RDS CAL token to a full RDS CAL token. If no license tokens are available, the temporary RDS CAL token will continue to function for 90 days.
When a client device receives an RDS Device CAL from an RDS Host, it receives it in the form of a digital certificate from a license server. That certificate is saved in the below location on Licensing server:

[HKLM\Software\Microsoft\TermServLicensing\Certificates]
[HKLM\System\CurrentControlSet\services\TermservLicensing\Parameters\Certificates.000]
[HKLM\System\CurrentControlSet\services\TermservLicensing\Parameters\Certificates.001]

The digital certificate is an actual certificate copied to the client device. Once a client device connects to an RDS Host, an RDS CAL digital certificate is transferred from the license server to the client device. The license server loses one of its licenses from its inventory, and the client device has the digital certificate that it can present to any RDS Host on future connections.

Clients store their license under the key:

[HKEY_LOCAL_MACHINE\Software\Microsoft\MSLicensing]

The MSLicensing key contains two sub-keys used to store both unique client-specific information and any license certificates obtained from license servers.

HardwareID
Store

HardwareID stores a Random 20-byte identifier specific to the client machine and is generated automatically by Windows. This ID uniquely identifies the machine to the license server. When a client is allocated an RDS CAL from the license server, this HardwareID is recorded in the licensing database to associate the client with the CAL. This entry is made when clients are allocated both temporary CALs and permanent licenses.

Store is used to store the terminal services CAL allocated from the license server.  Entries are contained in sub key named License00x, where X is a numerical ID beginning with 0.  Each License00x entry contains a separate CAL.

The License00x entry contains four binary components that comprise a terminal services CAL certificate:

• ClientLicense
• CompanyName
• LicenseScope
• ProductID

Every time the client device connects to an RDS Host, it presents its RDS CAL certificate to the server. The server checks not only whether the client device has a valid certificate, but also the expiration date of that certificate. If the expiration date of the certificate is within 7 days of the current date, the RDS Host connects to the license server to renew the license for another random period of 52 to 89 days.

Ideally each Client device should be issued only one RDS CAL. However, there would be times where License Server Manager will show multiple per device CALS being issues to the same device as shown in the below picture:

Now this is intriguing!! Why is the same device consuming multiple RDS CALS? The administrators usually notice this issue when they start running out of per device CALS and when they check the list of issued per device CALS in RDS licensing Manager, they notice that multiple RDS CALS have been issues to the same device.
To temporarily get around this issue you can revoke licenses but the catch is that you can only revoke 20% of the CALS at one time. This may not help if you have very few CALS left and you see that multiple per device CALS are being allocated to multiple machines.

Below are the possible reasons which can cause this issue:

1.    If you have built multiple machines using the same image:

a)    There could be times when you used a syspreped image or Citrix provisioned machines where the HardwareID was defined in the image because of which each device which was built using that image got the same hardware ID. This would result in the below situation:

• If Client1 has HWID xxxx and logs into the RDS, it will get license 1
• Then Client 2 which also has HWID xxx logs in and does not have license 1, so it's issued a new license, license 2
• If Server 1 tries to log in again, the xxx HWID is now associated with license 2, which Client 1 does not have, so Client 1 will get issued a new license, license 3
• Now the XXX HWID is associated with license 3
• Every time that HWID logs in, no matter what machine it is, its license will be compared to what's in the database for HWID XXX
• That's where the problem comes in -- machines are constantly getting new licenses, even when they aren't needed.

Resolution In order to get around this issue, you need to rectify the image itself and use a syspreped image which does not have MSLicensing Key information of the original machine hardcoded to it

b)    You Create a Citrix Provisioned machine where all the machines are booted from a pre-defined image and all the changes are lost after reboot. So every time the machine connects it gets a new ClientHWID and this is lost on the next boot. The next time the machine connects to the RDS Host, it gets a new Client HWID and hence a new RDS license is issued. Citrix XenDesktop provisioned machine with different hardware ID which can cause the license server to recognize it as different device and issue duplicate licenses

Resolution It is recommended to use Per-User RDS licensing in these scenarios, because the licenses are reverted when the user logs off, hence the number of licenses will not be affected.

2.    This could also happen if you have a script in place which deletes MSLicensing Key at shutdown.

Resolution Remove the script

3.    Different machines using same name.

If machines are cloned, sometimes third party cloning tools do not wipe out all the stale information and the cloned clients although with a different hardware it would give the same computer name to the RDS Host.

Though the Hardware ID might be different, if two different machines have the same name, looking at the Licensing Manager you might think that the same device is using multiple CALS but it is not.

4.    Machine was re-built:

For some reason if a machine that got a CAL once is re-built then due the new installation it got a new hardware ID and when connected again to Remote desktop server and hence got another CAL.

Assume that a client device successfully authenticates to an RDS Host and is granted a full RDS CAL certificate that was (worst case) randomly selected to expire at the 89 day maximum. When it passes down the certificate, the license server decrements its total RDS CAL license count by one, also noting that particular certificate's expiration date. Now, assume that a catastrophic event occurs at the client, causing its local operating system to be reinstalled and its local RDS CAL certificate to be lost. When that client authenticates to an RDS Host, the RDS will request a new RDS CAL certificate from the license server and the license server (again) decrements its RDS CAL inventory by one. At this point there have been two RDS CAL licenses given out to that one client, but the first one will never be renewed because the certificate was lost when the client was rebuilt. After 89 days (the randomly selected duration of the first certificate), the first RDS CAL is returned to the pool by the license server.

Resolution The old CAL will be freed within next 52-89 days after being issued or you can simply revoke the old CAL.

5.     Multiple Hardware ID’s in the MSLICENSING Reg key of the client machine:

This could happen if the license has been corrupted. If it has already been corrupted, a new hardware ID will be generated automatically for the client during next RDS Host logon and hence you may notice duplicate CALS for that device.

Resolution To determine which one you need to delete, go to the server, and open PowerShell “As Administrator” on the RDS License server, and execute the following command: get-wmiobject Win32_TSIssuedLicense | export-csv [outputfile]
Then in the output file, find out the client who is issued with multiple licenses, then record the hardware ID within the license which is not the most recently issued.
Then go back to the client, open registry, locate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSLicensing\HardwareID and check the ClientHWID which matches the one you just record, delete the HardwareID subkey.

DATA collection

1.    Look at TerminalServices-Licensing event logs.
2.    Generate per device RDS Per device Cal report to verify if the issue is because of multiple Hardware ID’s issued to the same machine, or same hardware ID issued to different machines or due to duplicate Machine names with different Hardware IDs.

Script for RDS Per Devices CALs (PowerShell)

This shows Keypack ID, License ID, Name of the client device along with Hardware ID and Expiration date of the CAL as shown below.

3. Use the RDS Client License Test tool (TSCTST.EXE) provided with the Windows Server 2003 Resource Kit on the client machine for which you see multiple CALS to display details about the license token residing on a client device. It is a command-line utility that displays the following information by default:

• Issuer
• Scope
• Issued to computer
• Issued to user
• License ID
• Type/Version
• Valid From
• Expires On

By using the /A switch, the following additional information is displayed:

• Server certificate version
• Licensed product version
• Hardware ID
• Client platform ID
• Company name

3. If you are still not able to find the cause, Microsoft professional can help you collect an RDS Licensing ETL trace while reproducing the issue. The etl trace should tell what name / HWID was used to request new licenses.

Quick Workarounds

1.    If all per device CALS are exhausted and you are working to find the case of multiple RDS CALS being issued to same device, temporarily you can change the licensing mode to per user to allow remote sessions. However, this should not be a practice as it will be a breach of Microsoft Licensing agreement.

2.    Regenerate the ClientHWID and Rebuild the License server database ( KB273566 ) and reinstall the CAL Packs to restore all the CALS.

The hardware ID can be regenerated by deleting the below keys manually:

Reg Delete HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSLicensing\HardwareID

Reg Delete HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSLicensing\Store /f

The next time you need to take an Remote session as an admin to regenerate the hardware ID as normal users do not have permissions on this registry key. Or you can use tools (RegenerateHDWID) to regenerate the hardware ID’s on the fly.

-Ishu