UI Elements from MRTK2.x on 2D/Flat Displays

[Deleted]

When you're developing a cross-platform experience, it would be great if MRTK made it possible (and easy) to adapt and use some of the UI elements from the MR version of your app/game, but in 2D.

By that, I mean... let's say you're developing a HoloLens 2 or Windows Mixed Reality VR game. You're using MRTK2.x for hand tracking on HoloLens and controller support in WMR. Your game's menu system is built using a Near Interaction menu. Now, let's assume you want to build a version of your game for Xbox, Windows 10, iOS, or Android.

On those platforms, it probably makes more sense to present the UI for your menu system in more of a traditional 2D way (flat, centered on the screen, overlayed on top of the game content) and using touch, mouse, or game controller for input.

It would be fantastic if you could just position the Near Interaction menu pane at the desired distance from the camera and have it respond to OnMouse events (just as an example) and with some visual feedback similar to what you get when your finger presses one of the buttons on HoloLens.

The hope being here, you can build one menu system and have it work on multiple platforms rather than have to implement something else entirely when your app is running a pancake screen device.

[#2]

This is something I think about often!

I'm rarely a fan of building things twice, and given how painful it has been for me to build UI in Unity, it's something I've never really gotten deep into. I've occasionally added quick flatscreen UI's using Unity's old immediate mode UI. It's not particularly pretty, but is much more simple and fun to build! But most of what I've worked on has been demos, and I could get away with a janky looking UI every now and again.

With StereoKit, I've got the opportunity to build something that does both, so I've thought about it a bit, but haven't tried to implement it yet. There is some overlap with far-interaction code that might make some of this technically easier, but it's tough to imagine a really good MR UI translating into 2D at all. Concepts like layout would need to be completely re-done, hand based UI would need re-thought, a lot of the 3D stuff becomes an obstacle rather than a benefit. I think in most cases, you'd need to re-implement the majority of the UI for flatscreen if you wanted to build a good experience. I have an upcoming personal project that has both flat and MR interfaces, and right now I don't plan on re-using even the conceptual design of the UI for that, they'll be completely separate. It definitely wouldn't work for that app.

But there's a chance MRTK already does some of this, or could be tweaked to do it anyhow? I know I've seen MRTK run on ARCore with touchscreens, and there's tools for getting Unity UI to work in MR as well. I poked the team with this post, we'll see if anyone drops by :)