MSIX packaging a dot net Core 3.1 app that references to windows.winmd assembly?

I have an winform app that targets dot net core 3.1. I have to use windows.winmd for AppserviceConnection or other necessary tasks.

While packaging the app with windows application packaging project (MSIX), running the WAP project gives an error:

The target process exited without raising a CoreCLR started event. Ensure that the target process is configured to use .NET Core. This may be expected if the target process did not run on .NET Core. The program '[0x63B4] WinFormsApp1.exe' has exited with code 2147516556 (0x8000808c).

Please Note:

1. Using Windows.Foundation.UniversalApiContract.winmd instead of windows.winmd also gives same error.
2. This winform project uses other assembly reference (System.Runtime.WindowsRuntime) which does not cause this error.

Is there any way to perform this action without error? (I can publish it as exe, but packaging with WAP project gives the mentioned error)