ReFS volume appears RAW (version doesn't match expected value) after Windows Update

[MikeLabatt]

After Windows Update last night, Windows Server 2019 wouldn't mount a storage space volume as ReFS (it appears as RAW). The error in the ReFS event log is "ReFS failed to mount the volume. Version 1.2 doesn't match expected value 3.4" No issues that I can see at the storage space level (it is a mirrored disk). The volume was working fine before Windows Update and the reboot. Another ReFS volume still works fine after the update.

Any clues? I could not find this error mentioned anywhere else. Thanks.

[#2]

Thanks. Tried this last week and it was painful to go back - Windows Updates does a good job only within a couple of weeks or so and going back to Dec 31st, 2021 hung my whole system and I had to go through system recovery to pick a more recent configuration.

Today I tried a different approach and restored from an old bare metal backup I had to a different drive, which took me to Win10 1803, then it updated to 20H2, where the problem still persisted and there was no KB's to uninstall. Much to my surprise, when I tried to uninstall that version, it didn't take me back to 1803, but rather dropped into 2004, unfortunately still with the problem, but then two KB's appeared - KB5010342 and KB5007401. I uninstalled them both and ReFS v1.2 got mounted. It was literally the last this I could try. No idea which one.

Can't believe Microsoft would do this to their users. They should have warned that support for ReFS v1.2 will be dropped every time it mounted and they had years for that. not a squeak. Moreover, why not release `refsutil` that can read v1.2, so people can run `refsutil salvage` against it to get their data back, even if it cannot be mounted. I wasted hundreds of dollars on recovery tools and efforts.

[#3]

All these tools scan drives for signatures and include deleted files and handle moved files extremely poorly. This turned my 4TB into 9.5TB one of the tools required. I had an intact volume and didn't need a scan, only for them to read the directory structure from v1.2. R-Studio was one of the worst - they couldn't even get a list of files working and their support had no idea. Other tools didn't even respond to support request, even after I paid for their tools. Microsoft should ether document ReFS publicly or provide much better set of recovery tools for these kind of things.

[#4]

If you data is still RAW after installing all updates and uninstalling the defective one first, then it is very likely that your ReFS volume has been formatted pre Windows Server 2016 and is not yet on ReFS version 3.x
ReFS version gets upgraded with OS Inplace upgrades when the volume remains attached (this cannot be reverted)
https://gist.github.com/0xbadfca11/da0598e47dd643d933dc

In this case have an old Server 2012 R2 Update 1 without this bugged patch and mount this volume to this server to gain access to your data with earlier REFS version.
If you then execute and inplace upgrade from 2012 R2 to 2019 this ReFS volume would be upgraded to the specific version Microsoft supports and the patch would work with.

Hope this helps.

[#5]

Agreed. Note however that refs v1 does not get auto updated to v3
That only happens on the v3.x versions ie 3.1>3.4->3.7
Hence why 'stuck' with older refs v1 volumes in some cases
ANother option is to to boot the 'Recovery OS' or boot from an Install iso /usb and select reapir. That will be a base OS version (without patches) and can access the data on refsv1 and copy it off.
Reminder - all this hassle only affects drives that are removable or have charactereistics of removable (like on vmware).
With the exception of Mikes original issue which is a weird and unexplained one.

[#6]

Thanks for clarification Stephen!
One question aside if a customer uses WinPE or WinRE from Win10/2016/2019/2022 is this one able to read ReFS natively, I am aware it is not for Storage Spaces.

Some still use Windows Backup to backup files this be restored? TY.

[#7]

Windows RE (Recovery OS) can read refs volumes ok

[#8]

I'd love to be the lone exception, but from other forums and Reddit discussions it seems I am not alone in having lost a ReFS 1.x volume that was not removable due to 2022 Windows Updates. It seems to me that AM-4566 and LarsKemmann in this thread were not using removable media either. I can understand it if Microsoft can only offer a workaround like "copy the data to a new volume", but not even admitting a problem on such a crucial technology seems a bit odd?

[#9]

@MikeLabatt @stephc_msft Stephen can you confirm that the remaining issue for ReFS 1.x is documented in the Windows Update History for affected Windows Server and Windows Clients? 

If not, I'd propose including it in Windows Update History and link to a new techcommunity article or KB article explaining the situation and solutions, so briefly what we discussed here.

If affected people with ReFS 1.x have a large amount files / data to copy instead of robocopy they can also leverage other tools. I would recommend Gurusquad GSCopy which is infact more scaleable, around 40% faster and has better reporting, open file support etc.

I regularly use this for file server migrations. 

If one want to stick with Microsoft Tools I would recommend to create a share on the root folder and use Windows Admin Center Storage Migration Service.
This is an easier process for those not being familiar with robocopy and it thas also good output. One needs to omit the last step to change servernames etc.

WAC SMS would only work if one creates a new machine (VM) with a ReFS 3.1 oder later volume. It is not designed to copy from one volume to another on the same machine.

[#10]

@stephc_msft I have a question about the following scenario

A customer had Windows Server 2019 VM attached to a VM controller and was affected by the ReFS RAW issue

The server had 2 ReFS volumes formatted with ReFS 3.1 4k Clustersize
Both were locally attached to a VMware controller of the VM (NVMe controller)

What happened:

One of the volume was affected on this machine the other was not affected.

You mentioned removable drives earlier so I wonder in which condition Windows considers SAS, SCSI or NVMe drives as removeable drives. 

Thanks for taking the time to elaborate on this scenario.

[#11]

For me, this has just reared its head when installing KB5014738 (2022-06 Security Monthly Quality Rollup). The exact symptoms as above, showing a RAW partition on an ReFS volume on Windows Server 2012 R2 on the Xen HV. I asked the HV host to make changes to the configuration of our VM to be xe vm-param-set uuid=<vm_uuid of the problematic vm> platform:device-model=qemu-trad but they refused so I have had to rollback the install of this KB.

No idea why this didn't impact us the first time round but just in case anyone comes across this thread and doesn't see their KB listed.

@stephc_msft Your suggested command to see the ReFS version number doesn't seem to work on Server 2012 R2. Is there a suitable command for this version instead of fsutil fsinfo refsinfo x:

[#12]

Thanks for the info on Xen HV.
WS2012R2 does not have the refsinfo option in fsutil, but doesnt matter too much, as if using ReFS on 2012R2 it will always be ReFS V1.2
If you can examine the first sector of the ReFS volume with a disk/hex editor then the version number is 'obvious' at offset 28 hex

[#13]

@stephc_msft God bless you sir! Just discovered this issue in July 2022 (the Exchange server affected had already had everything migrated to Exchange online so no end-user impact) and disabling hotplug in vCenter worked.

:clapping_hands::clapping_hands::clapping_hands::clapping_hands:

[#14]

Faced the same issue on Windows Server 2012 R2, disabling hotplug in vCenter worked :)

[#15]

@WHITE_ENG thanks for sharing your solution. can you please specify the Windows Server version that was affected and ReFS version by using fsutil fsinfo refsinfo AssignedVolumeLetter:

[#16]

Karl, the command to check the ReFS version does not work. For Server 2012 R2, it should be ReFS v1.

[#17]

yes that's correct and unlike to WS 2016 and later it won't be upgraded to 3.x
If you need to migrate tons of data efficient, I can recommend you to use Gurusquad GSCopy instead of robocopy or Windows Admin Center Storage Migration Service
Just in case you would upgrade from 2012R2 and want to relocate the data on a modern ReFS volume.
here are some of the noted benefits: https://gist.github.com/0xbadfca11/da0598e47dd643d933dc

[#18]

Hello, I believe I may be hitting this issue. We have a server 2019 guest vm on hyper-v cluster running Microsoft Azure Backup Server that we just built, have given it a pass-through SAN volume and when added to MABS it formats it with refs. After a short period of time, the volume now shows as raw in the vm and failover cluster manager. All of the hosts and the new vm have been updated, with the hosts at July's patch level and the vm at August's patch level. 

Any suggestions?

Thank you in advance!

[#19]

If it was formatted in WS2019 the should be ReFS v3.4 and should be ok, so shouldnt be that issue.
Are there any events in the ReFS event log ?

As a test can you temporarily connect that pass-through SAN disk to another VM that is at RTM or late 2021 build level and see if it can read it. If it can that could be related to the issue (but seems unlikely). If it also cant read it then , there is some real damage/corruption.

[#20]

@stephc_msft2 

Thank you for your response! as far as the logs, there is nothing consequential in the hosts, the VM however lists a lot of the following:

ReFS failed to mount the volume.
Context: 0xffff928b8960f180
Error: The I/O device reported an I/O error.

Volume GUID:{1eb14e22-8b1b-4ad6-bf40-bbb5038228f9}
DeviceName:
Volume Name:R:

I removed the volume from the VM and took it offline in failover cluster manager, now when trying to bring it back online it fails and looking in the storage pool area of server manager, it shows as operation status of disconnected. If I attach it to one of the hosts, I am then able to browse the contents, so the data appears intact.

It could be something MABS/DPM is having issues with. I would try the test you suggested, except I can longer bring the volume online within the cluster.

Thank you again for jumping in here, really appreciate it!

[#21]

@stephc_msft2
I created a new VM without updating it and a new passthrough SAN lun that is partitioned with refs. I'll let you know if it does the same thing.