Emerging Issue - Remote Access is disabled External Access Policy and NTLM is Disabled

[Sri Todi]

Emerging Issue - Remote Access is disabled External Access Policy and NTLM is Disabled

If legacy Authentication methods are turned-off externally by following https://docs.microsoft.com/en-us/skypeforbusiness/plan-your-deployment/modern-authentication/turn-on..., and remote access for the user is also disabled by External Access Policy a bug has emerged that causes clients on the external network to be in a infinite loop, trying to authenticate and get a 403 Forbidden error. This generally would happen whenever the client is not connected to VPN.

The bug manifests in many ways, some of which are mentioned below

1. Size of LCSCDR database can increase considerably, especially for the dbo.Registration table
2. CDR/QOE Reports may be delayed
3. In rare cases replication would show a single secondary as opposed to both active secondaries and would auto-correct after several hours

LYSS Database can experience an increase in size too, and you will notice

EVENT ID	Event id text	Notes
32056	Space Used by LYSS DB is within normal range	DB Utilization > 0% and < 40%
32057	Space Used by LYSS DB is at or above the Warning Threshold.	DB utilization  > =40% and < 60%
32059	Space Used by LYSS DB is at or above the Critical Threshold	Db Utilization is >= 60%

Depending on the extent of time the issue has been occurring, the size of the environment and other factors as user-behavior the following EVENT IDs may also be see

Event id	Event ID text	Notes
32075	A full flush of all queue items for LYSS DB has started.
32076	A full flush of all queue items for LYSS DB has completed.
32089	A flush of queue items from the LYSS DB was initiated, and items were exported to the file system.
32090	Flushed queue Items from the LYSS DB have been left unattended to for some amount of time and require attention to be imported back.
32103	Fabric service id 'ROUTING GROUP GUID' is running with a reduced replication set.	Get-CsPoolFabricState will show that routing groups are in missing secondaries

You can run a SQL Query against LYSS database to confirm if indeed you have been experiencing issues with 4003 by running

Use lyss;

SELECT     SUBSTRING ( CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])), CHARINDEX( '<MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader]))) + 10,  CHARINDEX( '</MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])))- (10+CHARINDEX( '<MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])))))  'MsDiag'     ,Count(1) 'Count'   FROM [lyss].[dbo].[ItemQueue]

WHERE CHARINDEX( '<MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader]))) > 0

Group by SUBSTRING ( CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])), CHARINDEX( '<MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader]))) + 10,  CHARINDEX( '</MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])))- (10+CHARINDEX( '<MsDiagId>', CONVERT(VARCHAR(MAX), CONVERT(VARBINARY(MAX), [ItemHeader])))))

Order by 2 desc

The output should look like

This issue has been fixed in a client update with version 16.0.11901.10000, but the default behavior hasn’t been updated. In-order to remediate the issue, you would need a client policy ( or a GPO) along with an updated client.

The fix to be effective we need a regkey ForbiddenRemoteAccessIsPermanentError as shown below.

Path:  HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Lync

KeyName: ForbiddenRemoteAccessIsPermanentError

Value: 1

The key can be pushed through client policy entry for e.g. adding the policy entry to global client policy

$a = New-CsClientPolicyEntry -name ForbiddenRemoteAccessIsPermanentError -value "True"

Set-CsClientPolicy -Identity Global -PolicyEntry @{Add=$a}

In-order for the client policy to be applied, a successful logon is required, so users need to sign-in atleast once, so the data is cached and used for subsequent failures

Once appropriate changes have been accomplished users will experience the following error message when logging on remotely

Please Note: At this point in time, only Skype for Business 2016 Client  has a fix, and there are no planned changes for Skype for Business 2015 Client

We understand that updating the clients may take some time, and while the clients are being updated, organizations may want a work-around to prevent any work disruptions. At this point in time, we are recommending the following

1. Update Storage Service behavior to disable Auto Import functionality to allow for a controlled method for import of data and prevent any potential issues by running

          Set-CsStorageServiceConfiguration -EnableAutoImportFlushedData $false
 

2. Perform a FULL Flush of storage service before the beginning of the day to prevent automatic export under load to happen during business hours, as it's resource-intensive ( CPU / Memory/ Disk / Network) by running
   Invoke-CsStorageServiceFlush -FlushType FullFlush -PoolFqdn POOLFQDN

This may also prevent FabricReplicationSetReduction happening in your organization, if it was previously occurring

Finally, it's possible that XML files have been written to your file share that may needed to be imported for regulatory and/or compliance purposes. Please reach out to Microsoft Support to help you find ways how/when the data can be imported safely.