This time I was involved to identify a problem that stuck all the DR tests in a large enterprise, let me show you a description of the environment.
The Customer have a Production Hyper-V Farm (multiple HV clusters) on a Primary Datacenter, this farm have multiple LUNs (Logical Unit Number) on a Storage visible to the hosts via FC (Fiber Channel). The storage replicate the LUNs with an asynchronous replica to the DR (Disaster Recovery) datacenter.
The customer, during the year, make multiple simulation on the DR datacenter to validate if the DR work correctly, but in the last simulation the Orchestrator server in the DR datacenter stopped the execution of the runbooks, that automate the import of the VMs from the LUNs, with the following error message:
"Add-ClusterVirtualMachineRole : The File exists."
TROUBLESHOOTING We identified inside of the orchestrator runbooks the piece of code that generate the exception and we retried the execution of the cmdlet only on a single VM like in this case:
We start thinking that may be the VM was already present in the Hyper-V cluster, but why we receive the message "The File Exists"! So we have retried the command with the "-verbose" parameter, and we received this additional info:
From this info, it seams that the cmdlet "Add-ClusterVirtualMachineRole" (that add the VM to the Hyper-V cluster) is trying to get a temporary file:
VERBOSE: Connecting to cluster on local computer HV1-2016.
VERBOSE: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
Because the customer during the year have made multiple tests on the DR environment, the Orchestrator user that start the runbooks had more than 65535 temporary files in the temp user folder with this kind of syntax "tmpXXXX.tmp".
The final solution was to add few line of code at the beginning of the runbook on the orchestrator server to cleanup the temp folder of the orchestrator user.
NICE TO KNOW
I was able to reproduce the Customer issue on a single Hyper-V Cluster in my lab, by using the "New-TemporaryFile" cmdlet, because it use the same method for the creation of the TMP files.