Forum Discussion
Weird Storage Migration Service Error (Failing after 45gb)
I haven't been able to find any other instances of this error so hopefully someone here might have (or a helpful MS employee might be able to assist).
Trying to test a file server migration from 2012 R2 to a new Server 2022 VM, I've had Windows Admin Center running in a dedicated Server 2022 VM and running on the new file server itself and both hit the same error. I've removed any 3rd party AV software from all servers concerned just incase it was interfering but this didn't help either.
I can get the job created and it starts the transfer process but after around 45gb or 23k files it errors with the SMS job showing "Couldn't transfer storage on any of the endpoints".
The Transfer Detail error log is empty so it doesn't look like it is failing due to file permissions (I've had this before and fixed them).
However in the SMS Proxy debug event log there are errors shown right when the job fails:
10/26/2022-10:21:06.708 [Erro] Failed pipeline execution. System.AggregateException: One or more errors occurred. ---> System.ArgumentOutOfRangeException: Index and count must refer to a location within the string.
Parameter name: count
at System.String.Remove(Int32 startIndex, Int32 count)
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferUtil.CreateErrorTransferInformation(FileInfo info, String pathRoot)
at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.TryTransfer()
at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransferStage.ProcessItem(RetryInfo retryInfo)
at Microsoft.StorageMigration.Proxy.Service.Transfer.StageBase`3.DoStage(CancellationTokenSource cts)
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.StorageMigration.Proxy.Service.Transfer.Pipeline.Run(CancellationToken token)
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferOperation.Run()
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferRequestHandler.ProcessRequest(FileTransferRequest fileTransferRequest, Guid operationId)
---> (Inner Exception #0) System.ArgumentOutOfRangeException: Index and count must refer to a location within the string.
Parameter name: count
at System.String.Remove(Int32 startIndex, Int32 count)
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferUtil.CreateErrorTransferInformation(FileInfo info, String pathRoot)
at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.TryTransfer()
at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransferStage.ProcessItem(RetryInfo retryInfo)
at Microsoft.StorageMigration.Proxy.Service.Transfer.StageBase`3.DoStage(CancellationTokenSource cts)
at System.Threading.Tasks.Task.Execute()<---
[d:\os\src\base\dms\proxy\transfer\transferproxy\TransferRequestHandler.cs::ProcessRequest::132]
and
10/26/2022-10:21:06.708 [Crit] Consumer Task failed with error:One or more errors occurred.. StackTrace= at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.StorageMigration.Proxy.Service.Transfer.Pipeline.Run(CancellationToken token)
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferOperation.Run()
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferRequestHandler.ProcessRequest(FileTransferRequest fileTransferRequest, Guid operationId)
at Microsoft.StorageMigration.Proxy.Service.Transfer.TransferRequestHandler.ProcessRequest(Object request)
at Microsoft.StorageMigration.Proxy.Common.ProducerConsumerManager`3.Consume(CancellationToken token) [d:\os\src\base\dms\proxy\common\proxycommon\ProducerConsumer.cs::Consume::117]
I'm pretty much stumped at this point
So interestingly I was able to figure this one out after a few more hours of digging around, in this case it was going back through the SMS-Proxy debug eventlog as this contained a log for each file copied I was able to trace the errors back to the file that triggered it all:
10/26/2022-10:21:06.511 [Erro] Transfer error for \\server.domain.com\E$\Data\Share\Folder\CON.CSV.CSV: (2) Failed to get file handle. Stack Trace: at Microsoft.StorageMigration.Common.FileDirUtils.OpenFile(String fileName, DesiredAccess desiredAccess, ShareMode shareMode, CreationDisposition creationDisposition, FlagsAndAttributes flagsAndAttributes) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.StreamTransferInfo.OpenTargetHandle(String fileName, Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.OpenStreamHandles(Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.InitializeDataStreams(Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.CopyAllStreams() at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.StartCopy() at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngineBase.CopyData() at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.Transfer() at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.TryTransfer() [d:\os\src\base\dms\proxy\transfer\transferproxy\FileTransfer.cs::TryTransfer::63]Despite the Transfer Error log in SMS being empty it does seem like it was a file that caused the issue all along since CON is a reserved "special" name it looks like this was tripping SMS up (this particular folder gets data uploaded into it via FTP from our ERP system which explains how the filename was created I think).
https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#file-and-directory-names
The fix here was to rename the file, trying to rename it from the server directly errored but renaming via a network share from another server worked fine, I was then able to restart the SMS transfer and it worked (I've still got a few more permission errors to fix but that is easy enough compared to this!).
So interestingly I was able to figure this one out after a few more hours of digging around, in this case it was going back through the SMS-Proxy debug eventlog as this contained a log for each file copied I was able to trace the errors back to the file that triggered it all:
10/26/2022-10:21:06.511 [Erro] Transfer error for \\server.domain.com\E$\Data\Share\Folder\CON.CSV.CSV: (2) Failed to get file handle. Stack Trace: at Microsoft.StorageMigration.Common.FileDirUtils.OpenFile(String fileName, DesiredAccess desiredAccess, ShareMode shareMode, CreationDisposition creationDisposition, FlagsAndAttributes flagsAndAttributes) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.StreamTransferInfo.OpenTargetHandle(String fileName, Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.OpenStreamHandles(Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.InitializeDataStreams(Boolean openForReadonly) at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.CopyAllStreams() at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngine.StartCopy() at Microsoft.StorageMigration.Proxy.Service.Transfer.CopyEngineBase.CopyData() at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.Transfer() at Microsoft.StorageMigration.Proxy.Service.Transfer.FileTransfer.TryTransfer() [d:\os\src\base\dms\proxy\transfer\transferproxy\FileTransfer.cs::TryTransfer::63]Despite the Transfer Error log in SMS being empty it does seem like it was a file that caused the issue all along since CON is a reserved "special" name it looks like this was tripping SMS up (this particular folder gets data uploaded into it via FTP from our ERP system which explains how the filename was created I think).
https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#file-and-directory-names
The fix here was to rename the file, trying to rename it from the server directly errored but renaming via a network share from another server worked fine, I was then able to restart the SMS transfer and it worked (I've still got a few more permission errors to fix but that is easy enough compared to this!).
