Hello,
We have some xUnit tests that we are trying to run in an ADO Release Pipeline. We have associated these xUnit tests with Test Cases, and the goal is to run these test cases from the Test Plan and trigger these xUnit tests via the Run with Options > Automated tests using release stage (in Release Pipeline).
We have found that these xUnit tests run fine in a 'conventional' CI-Pipeline using Yaml to trigger dotnet test. However, when we attempt to run the exact same dll but from the Release Stage pipeline, the dll associated with the test case, and the test method are seen in the logs, and it is indicating that it is looking in the correct path, but for whatever reason it doesn't execute the tests and says it cannot find them.
On github I observed a bug (#11607) that had been closed that appeared to document the same problem. I have since opened another bug (#18596) . Through my company I contacted support, but they recommended to post the issue here to try to get resolution or a workaround.
The logs show the following:
2023-07-06T18:41:39.9486663Z List of test cases in slice with id: 65
2023-07-06T18:41:39.9487835Z Source: RanorexVerification.dll, FullyQualifiedTestName: Gemini_Ranorex_137970
2023-07-06T18:41:39.9488615Z ##[debug]PERF: Execution:Run: took 2.3094 ms
2023-07-06T18:41:39.9992021Z ##[error]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-07-06T18:41:39.9993699Z ##[debug]Processed: ##vso[task.logissue type=error;]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-07-06T18:41:39.9995448Z at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-07-06T18:41:39.9998064Z at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-07-06T18:41:39.9999171Z ##[debug]ExecuteAndPublish.Run : Cleanup invoked.
The release pipeline is identifying the correct dll and FullyQualifiedTestName from the automation associated with the test case. I've also confirmed that it is searching the correct path for the dll, which I have confirmed exists on the test VM as seen in these earlier messages in the log:
Build location: C:\RanorexVerification\RanorexXunit\RanorexXunit
2023-07-06T18:41:28.5418662Z Build Id: 74324
2023-07-06T18:41:28.5419314Z ##[debug]Current working directory: c:\Agent\_work\r1\a; Test drop location: C:\RanorexVerification\RanorexXunit\RanorexXunit
Also, as previously stated, this dll runs just fine from a yaml pipeline executing dotnet test on the exact same VM, so the problem appears to stem from some issue with Release Pipelines.
Any help would be greatly appreciated!
