2210 HF4 still broken on Server 2019 and new Teams

[Ken_Z]

Hi Everyone,

so, I assume everyone's been testing the new 2210 HF4 this morning/last night?

I've got two Citrix Session Hosts, Server 2022 and Server 2019

Both of them have the latest FSLogix installed, and they share the same GPO for the FSLogix configuration. 

They've also got a common redirections.xml file for both Servers. Logging into a Session host creates both a normal profile folder and a local_<user> folder for redirected folders/files

For testing, I've been deleting the vhdx files between tests on each server, so there's no potential issues using a profile container between the two different OSes.

Also, I've only got a single profile container configured - I'm not using an Office container

I've also got the latest new Teams installed on both servers. Server 2022 was installed via the bootstrapper and Server 2019 was installed via DISM as per the recommended method.

Server 2022 - when i log in to this OS using a test account configured for FSLogix, and start new Teams, everything is fine. When i log off, both the user profile and local_<user> folders get deleted.

Server 2019 - Doing exactly the same operation on this server, after starting new Teams, then logging off, the local_<user> folder fails to get deleted. The error message in the FSLogix operational logs are "Error removing directory: C:\users\local_<xxxx>\AppData\Local\Microsoft\IdentityCache\1\UD\.... The process cannot access the file because it is being used by another process)"

Note that the location under AppData\Local\... can change between different tests as there are a lot of different files locked and this is the first file FSLogix came across during the logoff process.

Running sysinternals Handle64.exe shows that the files are open by "System" (pid = 4)

Also, I can't delete the folders manually as administrator after the user has logged off either, unless i reboot the server.

This issue only occurs after I've run new Teams for the first time. If i start clean without a profile container, log in, start Word, Excel, Outlook, etc, but don't start new Teams, the issue doesn't occur. Once I've started Teams, even if i shut it down, the local_<user> folder can't be removed/deleted.

Anyone else experiencing this?

[#2]

I should mention that i was also getting this exact behaviour with HF3 as well, but was hoping that HF4 would fix it. A customer that has Server 2019 RDS is also experiencing the exact same issue with HF3,

I also tried completely removing FSLogix, deleting all the regkeys under HKLM\SOFTWARE\FSLogix and HKLM\SOFTWARE\Policies\FSLogix, then installed HF4 as a clean install, and the same issue occurred.

The issue does not occur with Classic Teams

[#3]

I also struggled over this but then i checked my FSLogix Installation with frxtray.exe

The Version that should be running is:

FSLogix Service Version 2.9.8884.27471 Status: RUNNING
FSLogix Kernel Driver Version 2.9.8884.27471 Status: RUNNING
FSLogix Kernel Virtualization Driver Version 2.9.8884.27471 Status: RUNNING

but why is it running the old Version 2.9.8784.63912 after the Update?

So i uninstall FSLogix and make sure that is no FSLogix in C:\Program Files and in ProgramData.

Reboot the Server and install the new Version of FSLogix and reboot again.

Now the new Version 2.9.8884.27471 is running checked with frxtray.exe

Hope this will help.

[#4]

Hi Hans
I'd completely uninstalled the old version before installing the new one, but I've just re-checked anyway. Under Driver Interface i see

Profile Version: 6
Installed system RAM: 8191 MB (load: 42 %)
FSLogix Service Version 2.9.8884.27471 Status: RUNNING
FSLogix Kernel Driver Version 2.9.8884.27471 Status: RUNNING
FSLogix Kernel Virtualization Driver Version 2.9.8884.27471 Status: RUNNING

so it's all at the correct version.

Regards

Ken Z

[#5]

I've logged this with Microsoft today. The engineer who I spoke to says he's seen already seen this! and it's only ben 24 hours since HF4 came out.

[#6 · Accepted solution]

OK, if anyone is interested, I got to the bottom of this issue. I had to remove the following entries from my redirections.xml file as New Teams was causing the system to keep the files in these folders locked after logoff. (This only occurs on Server 2019, not Server 2022)

<Exclude Copy="0">AppData\LocalLow</Exclude>
<Exclude Copy="0">AppData\Local\D3DSCache</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\IdentityCache</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\OneAuth</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\TokenBroker</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\Windows\Caches</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\Windows\Explorer</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\Windows\IECompatCache</Exclude>
<Exclude Copy="0">AppData\Local\Microsoft\Windows\iecompatuaCache</Exclude>

Once these were removed, the local_<user> folder was successfully being deleted.
NOTE this is my standard xml file that i've been using for years and hasworked without incident, and still works on Server 2022. Definitely an issue with New Teams on Server 2019.

Note the LocalLow was removed but it was only a couple of files in the Microsoft subfolder that was causing the issue.

[#7]

@Ken_Z 

Same issue here on server 2019 + fslogix + new teams on RDS Server.

C:\users\Local_user1,2,3 keeps adding up numbers, user folders not deleted when signing off.

fslogix log show:

Error removing reparse point: C:\User\local_User\AppData\Local\Pacakges\MSTeams+8wekyb3d8bbwe\LocalCache 

(process cannot access the file, because another process already uses it)

any ideas?

I have tried you suggestion and changed the redirections.xml not working.

[#8]

hjmmm

have you rebooted the server after changing the redirections.xml file?
also, to confirm that it's the same issue, try completely removing the xml file from a test server so that it doesn't exclude any files, delete any vhdx files from your test account, and retry.
if the issue goes away, most likely you've missed an entry to remove.

Ken Z

[#9]

Seems to work with new vhdx and no xml, when re enable the xml i get error: Error removing directory C:\users\local_user\AppData\Local\Packages\Microsoft.AAD.BrokerPlugin_\AC\Microsoft\CryptnetUrcl\Cache (folder not empty)

[#10]

so it looks like one of the Excludes is causing the issue. how big is the excludes list? can to share the XML file here?

[#11]

@Ken_Z 

<Excludes>

<Exclude Copy="0">Videos</Exclude>

<Exclude Copy="0">Saved Games</Exclude>

<Exclude Copy="0">Contacts</Exclude>

<Exclude Copy="0">Searches</Exclude>

<Exclude Copy="0">Citrix</Exclude>

<Exclude Copy="0">Tracing</Exclude>

<Exclude Copy="0">Music</Exclude>

<Exclude Copy="0">$Recycle.Bin</Exclude>

<Exclude Copy="0">AppData\LocalLow\Adobe</Exclude>

<Exclude Copy="0">AppData\Local\Apps</Exclude>

<Exclude Copy="0">AppData\Local\Downloaded Installations</Exclude>

<Exclude Copy="0">AppData\Local\assembly</Exclude>

<Exclude Copy="0">AppData\Local\CEF</Exclude>

<Exclude Copy="0">AppData\Local\Comms</Exclude>

<Exclude Copy="0">AppData\Local\Deployment</Exclude>

<Exclude Copy="0">AppData\Local\FSLogix</Exclude>

<Exclude Copy="0">AppData\Local\Packages</Exclude>

<Exclude Copy="0">AppData\Local\VirtualStore</Exclude>

<Exclude Copy="0">AppData\Local\CrashDumps</Exclude>

<Exclude Copy="0">AppData\Local\Package Cache</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Notifications</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Internet Explorer\DOMStore</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Internet Explorer\Recovery</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\MSOIdentityCRL\Tracing</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Messenger</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Terminal Server Client</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\UEV</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\Application Shortcuts</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\Mail</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\WebCache</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\WebCache.old</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\AppCache</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\GameExplorer</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\DNTException</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\Notifications</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\PRICache</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\PrivacIE</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\RoamingTiles</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\SchCache</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\Temporary Internet Files</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\0030</Exclude>

<Exclude Copy="0">AppData\Local\Microsoft\Windows\1031</Exclude>

<Exclude Copy="0">AppData\Roaming\com.adobe.formscentral.FormsCentralForAcrobat</Exclude>

<Exclude Copy="0">AppData\Roaming\Adobe\Acrobat\DC</Exclude>

<Exclude Copy="0">AppData\Roaming\Adobe\SLData</Exclude>

<Exclude Copy="0">AppData\Roaming\Microsoft\Document Building Blocks</Exclude>

<Exclude Copy="0">AppData\Roaming\Microsoft\Windows\Network Shortcuts</Exclude>

<Exclude Copy="0">AppData\Roaming\Microsoft\Windows\Printer Shortcuts</Exclude>

<Exclude Copy="0">AppData\Roaming\ICAClient\Cache</Exclude>

<ExcludeCopy="0">AppData\Roaming\Macromedia\FlashPlayer\macromedia.com\support\flashplayer</Exclude>
</Excludes>

[#12]

actually, just noticed that the file/folder locked is the Microsoft.AAD.BrokerPlugin_ package.
Are you using any type of Legacy Azure MFA /conditional access policies) in your environment?
Also, exactly how did you install the new Teams, and did you uninstall the old Teams first?

[#13]

and can you also show your include list too please

[#14]

Installed new teams with "Dism /Online /Add-ProvisionedAppxPackage /PackagePath:c:\temp\MSTeams-x64.msix /SkipLicense"

old error is back now:
Error removing reparse point: C:\User\local_User\AppData\Local\Pacakges\MSTeams+8wekyb3d8bbwe\LocalCache (no access)

[#15]

No includes

[#16]

Hjmmm

that's why it's not working. The official documentation says you must add the following into the include section of the redirections.xml file

<Includes>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\LocalCache\Microsoft\MSTeams</Include>
<Include>AppData\Local\Publishers\8wekyb3d8bbwe\TeamsSharedConfig</Include>
</Includes>

read https://learn.microsoft.com/en-us/microsoftteams/new-teams-vdi-requirements-deploy

[#17]

Looks like the above article has been updated since i originally read it and there's more includes now. I've updated my includes to look like this

<Includes>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\AC</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\AppData</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\LocalCache</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\LocalState</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\RoamingState</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\Settings</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\SystemAppData</Include>
<Include>AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\TempState</Include>
<Include>AppData\Local\Publishers\8wekyb3d8bbwe\TeamsSharedConfig</Include>
</Includes>

I'll test it out to see how it behaves

Regards

Ken

[#18]

I've include the 2 lines, now teams is not starting in session. Looks like getting installed when click on teams but teams is not starting.

[#19]

Hjmmm

OK, tested this on my Server 2019 Citrix 2402 LTSR Session Host running FSLogix 2.9.8884, WebView2 Runtime 126.0.2592.113 and Teams 24165.1408.2960.3667

Deleted any existing FSLogix container for my test account
Logged in
Started Teams
Sent a teams message to another account.
Checked that both C:\users\<user> and C:\users\local_<user> existed on the session host
Logged off.
both C:\users\<user> and C:\isers\local_<user> got deleted successfully
logged in a second time and repeated the process
Logged off again and both folders successfully removed

Regards

Ken Z

[#20]

hjmmm

firstly, can you confirm the versions of FSLogix, Teams and WebView2 Runtime that you've got installed.
Also, have you deleted the existing profile container before doing the test? the redirections.xml file gets cached in the FSLogix container and sometimes it doesn't get read properly between sessions, so deleting the container ensures that the latest xml file gets used.
Finally, make sure that any C:\Users\local_<user> folders are deleted manually between tests and if necessary, reboot the test server between tests.

If none of the above helps, can you can send me your email address and I'll send you my entire redirections.xml file for you to try.

regards

Ken Z