MRTK3 Public Preview is now available on the MR Feature Tool as well as GitHub! MRTK3 offers many improvements with its architecture, UI tools, interaction system, and other new features.
- Download MRTK3 Public Preview
- Learn more: Introducing MRTK3 – Shaping the future of the MR Developer Experience
- Join us on July 14th, 2022: MRTK3 Public Preview Ask us Anything session
Modular Packaging
We’ve separated MRTK into many modular packages that you can independently install and update. You can pick and choose the MRTK packages that suit your needs, significantly reducing the amount of MRTK code needed to take advantage of our features.
MRTK3 is all-in on OpenXR and Unity’s XRI Toolkit. One of the goals with MRTK3 was to take everything we've learned from the start of MRTK2 back in early 2018, combine it with the work that's been done by our industry partners across OpenXR and Unity since then, and come out the other side with a strong, extensible foundation that allows MRTK to focus more on providing differentiators and an overall improved user (and developer!) experience.
OpenXR
OpenXR is an open royalty-free API standard from Khronos, providing engines with native access to a range of devices across the mixed reality spectrum. Unity ships an OpenXR plugin providing integration of the core spec for features like rendering, head tracking, and controller input. Microsoft ships the Mixed Reality OpenXR Plugin, which builds on Unity’s plugin and adds support for additional features relevant to devices like HoloLens 2. Microsoft-specific additions include spatial mapping and planes, spatial anchors, locatable camera, and tracked hand mesh. It also adds support for hand joint tracking via a cross-vendor OpenXR extension, which means it’s supported on more devices than just Microsoft’s. This is the piece MRTK3 depends on for features like articulated hand joint tracking and the poke pointer.
- Learn more: #Open - Deploy Everywhere with OpenXR and MRTK3
XR Interaction Toolkit
MRTK3 uses the Unity XR Interaction Toolkit (XRI) as the foundation for input abstractions, interaction, and manipulation. Most of MRTK3 is simply a very thin layer atop the interaction primitives provided by XRI; as a result, MRTK3 is no longer a bespoke interaction and input system, but merely a well-behaved “citizen” of the XRI framework and community. In fact, Unity’s XRI was built with the MRTK team’s input, and we’ve been fortunate enough to have frequent syncs with the XRI team to give feedback, help with bugs, and ensure that the MRTK systems are built in the spirit of the XRI framework. We’ve also been able to help shape what XRI looks like in terms of future plans and architectural decisions.
One of the biggest advantages of this approach is that MRTK3 is highly compatible with other XR applications and interaction systems, if they also build on top of XRI. For instance, existing unmodified XRI projects and applications can utilize MRTK3 interactables like volumetric UI controls, ObjectManipulator, and BoundsControl. On the other hand, MRTK3 projects can use existing XRI-based interactables, such as those that ship out of the box with the XRI framework itself. Even the XR rig itself is highly flexible and compatible, and custom rigs can be used with no MRTK code whatsoever.
Most importantly, we can rely on the rock-solid interaction systems that XRI provides, and focus on what MRTK does best, like new ways to manipulate holograms, collections of useful prefabs, volumetric UI/UX, and other mixed-reality-specific building blocks. Encouragingly, XRI has already seen wide adoption across the industry and has generated a fair bit of excitement in the XR space. We’re proud to be working with Unity on this and we’re excited to see what opportunities this collaboration will unlock, especially as we expand MRTK’s out-of-the-box compatibility to a much wider range of XR applications.
Volumetric UI on Canvas
MRTK3 introduces volumetric UI integrated with Unity's RectTransform + Canvas system. While RectTransform and Canvas have historically been primarily used for 2D flat UI, they’re also fully capable of rendering and laying out volumetric 3D UI. MRTK3 includes systems and prefabs for full-parity canvas-based 3D UI, accelerating design iteration and raising the bar for fidelity and detail when designing volumetric user interfaces.
Previously in MRTK2, building UI was difficult, especially if any dynamic layout or collections were required. Building on Unity’s RectTransform-based UI tooling allows for a completely re-invented UI workflow. Unity’s layout groups can be used for dynamic and flexible arrangements of content within your UI, in the same way that they are used for traditional flat UI. In addition, many pieces of basic functionality that designers expect from modern design tools like resizing, responsive layouts, alignment, anchoring, margin, and padding are all included by default.
Instead of building UI manually with hand-calculated dimensions, lossy scaling operations, and fragile static layouts, UI can be constructed much in the same way that designers use other modern presentation frameworks. Layouts are built from horizontal, vertical, and grid-like groups, and are fully composable and responsive. The sizes of panels, windows, and slates can be inflated from a dynamic number of child elements, and the children can be edited, reordered, and removed at runtime as the layout system responds immediately.
Another convenient benefit is that by unifying our UX controls under the Canvas/UnityUI system, we can take advantage of Unity’s cross-platform UI input system to enable directional navigation and gamepad input by default. Out of the box, you’ll get rich gamepad support on the same volumetric UI controls that you can touch with hand-tracking. It’s also a great win for accessibility, because many accessibility inputs and controllers require these traditional 2D gamepad-type input modalities. Architecturally, these traditional 2D inputs (including gamepad, mouse, touchscreen, and more) invoke the exact same events and appear completely identical to advanced XR interactions like poke and gaze-pinch, which means that there’s a single codepath for every single method of interaction with your UI.
As a result, building production-grade UI is vastly easier and more maintainable, and can be scaled across larger teams and more mature designs. It’s been critical for us internally, as we build larger and more complex mixed reality applications, we’ve needed more mature, robust, and maintainable UI systems. In addition, it’s allowed us to reduce the quantity and churn of prefab variants, as we no longer need a huge number of prefabs for each permutation of UI control size or configuration. These improvements have let much larger teams work together while keeping our design language consistent and polished across large and complex layouts.
- Learn more: Building Rich UI for MR in MRTK3
New Mixed Reality Design Language
In MRTK3, we are providing refreshed UI building blocks based on the new design language that we introduced in the Mesh app for HoloLens 2 last year. Throughout the past several years, the team has learned a lot from many different projects and experiments; the new Mixed Reality Design Language is the result of many design iterations by our designers, researchers, and engineers, and we hope you’ll love it.
Here are some of the updates:
- Rounded corner geometry for more approachable and friendly experiences
- Updated visual system (grids and modules) to support various types of Mixed Reality UI scenarios
- Improved visual feedback for multi-modal input such as eye-gaze with pinch gesture input
- Modular backplate system for building complex layouts that remain clear and usable
- Updated bounding box visuals to reduce visual noise and enable fluid gaze-powered interactions
Examples of modular UI building blocks and improved visual feedback for multi-modal input
Advanced Interactions
Gaze in MRTK3
MRTK3 aims to make it as easy as possible for users to use gaze to interact with objects. To accomplish this, we’ve dramatically improved our gaze targeting implementation, as well as introduced a new kind of first-class interaction type; gaze-pinch manipulation.
Gaze targeting improvements:
Throughout most of MRTK2, gaze targeting was a more limited feature. You could adjust the minimum and maximum gaze distance, and the user’s eye gaze was captured as a single raycast which followed the user’s tracked eye gaze. While this implementation was functional, it had difficulties targeting smaller objects and required users to keep a focused, steady gaze to maintain targeting.
In MRTK3, we’ve dramatically improved this implementation. The user’s eye gaze is now represented by several spherecasts, each of increasing precision. These coverage of the spherecasts can then be further refined to a cone by specifying a cone angle. This allows objects which are not directly aligned with the user’s eye gaze to be targeted, greatly reducing the effort required to target and maintain targeting on smaller objects. In conjunction with this, we’ve also developed a smarter algorithm which scores all potential targets captured by the user's eye gaze, using different factors like distance and angle to determine what the user is most likely trying to select.
All of these parameters can be tweaked by a developed at design-time and run-time, providing developers with the flexibility to tune gaze in the way that best suits their needs.
Gaze-Pinch
Eye gaze isn’t just a great way to passively target and select objects; it’s also a great way to determine the user’s intent and focus, and is especially powerful when paired with hand-tracking-based manipulation gestures. We’ve developed a delightful and powerful new way for users to manipulate objects through a combination of gaze targeting and subtler hand gestures, aiming to reduce both mental and physical fatigue during interactions.
Gaze-pinch leverages the new fuzzy gaze algorithm described above to first determine which object the user intends to manipulate. Then, then user can use their fingers and hands to manipulate the object as if it were nearby, using pinch, grab, and other familiar input gestures, even while the object may be far out of reach. In this input modality, we can relax the constraints of what we consider an “upright hand” to be, enabling subtler hand manipulations (instead of pointing a ray at the object!)
Dwell-based gaze interactions, like you might be familiar with from MRTK2, make a return in MRTK3. You can still use the same dwell-based interactions as you did in previous versions of MRTK, even alongside the new gaze-pinch system. Dwell is a great solution for enabling more accessible UI in your app, and it’s even better in MRTK3 with our improvements to the gaze detection system.
Multi-handed by default
In MRTK2, users enjoyed intuitive multi-handed interactions, both when manipulating holograms directly and when using hand rays. We’ve done quite a lot of interaction design iteration to perfect multi-handed interactions, and we’re glad to have been able to share our work in the past in MRTK2.
In MR environments, users expect to be able to use both of their hands at minimum, making multi-input interaction more important than ever. In MRTK3 all interactables can now be interacted by multiple interactors by default, thanks to our collaboration with Unity’s XRI framework. Using XRI’s clearly defined interaction states and events, developers can design interactables with multi-input in mind. This means a button can now be pressed by multiple input sources and can remain pressed when any of those input sources ends its selection.
In MRTK3, we’ve taken multi-handed interactions to the next level. Friendly APIs for dealing with an arbitrary number of participating interactors make it easy to build more complex multi-handed interactions. In MRTK2, multi-handed interactions were mostly used with ObjectManipulator and BoundsControl; now, it’s included by default in every interactable, and users can expect all objects to react reasonably to any number of inputs, simultaneously. Buttons, sliders, and even drawing surfaces respond to an arbitrary number of interactions, without requiring the developer to keep track of each input manually.
Variable Selectedness
Mixed reality is fundamentally a highly “analog” input scenario; most ways that users can interact with holograms are variable, analog, or even sometimes subjectively determined from the user’s body. As a result, we are faced with a set of highly “fuzzy” inputs from the user, along with a collection of interactables and behaviors that must respond to these variable inputs. Motivated by this challenge, MRTK3 introduces the concept of variable selectedness, allowing users to select objects partially and with a wide range of input modalities, such as partial button presses, partial pinches, analog triggers on controllers, and more.
Internally, our interactables have a selectedness variable, which is a max of the selectedness returned by all hovering and selecting interactors. For interactors which implement IVariableSelectInteractor, this can be calculated using pinch strength, trigger-pressedness, or whatever the developer decides to implement. For all other interactors, it is either 0 or 1 based on whether the interactor is currently selecting or not.
This definition of selectedness is fully extensible, as more specialized interactables like our buttons also incorporate the pressedness of a touch interactor to determine its selectedness value. However, no matter what interactor you decide to use, our MRTK3 interactable will be able to respond to it, without needing to change any of its underlying source code.
Interaction Modes
In MR, users have multiple ways to interact with an object, no longer restricted to a singular mouse pointer/finger/controller. To best leverage the advantages having multiple “controllers” gives people, MRTK3 introduces the concept of Interaction Modes. Each interaction mode specifies a set of Interactors that are active, and each controller is in a single interaction mode at any given time. This allows the developer the flexibility to allow each controller to be in a different interaction mode or to synchronize them all to the same mode. This can be very useful for cases where the user may need to manipulate a nearby object with one hand, while targeting a far away object with another.
Modes will take priority over each other based on the order they are defined in the Interaction Mode Manager. This makes the assignment of a mode via a Mode Detector a monotonic operation, no matter what order the mode changes are specified in, the mode with the highest priority will always be assigned to the relevant controllers in the end.
- Learn more: MRTK3 Interaction Building Blocks
Data Binding
In MRTK3, we provide tools that make it easier to build dynamic UI. We provide a framework for sourcing, binding, and consuming a wide variety of dynamic data; this includes binding text to your UI, updating materials and textures on the fly, and even binding more complex data like audio clips. Data and updates can be pulled from any type of source, including both remote web services and locally-hosted JSON or assets.
Although we still have quite a few improvements planned for data binding prior to MRTK3 General Availability (GA), it is already in production in Dynamics 365 Guides v7 released last fall. Here are buttons and lists that are all populated using the new data binding feature:
Theming
MRTK3 Theming makes it possible to change the entire look and feel of an entire application, scene, or prefab at runtime. The theme’s styling information can be specified both locally and remotely, either through a configuration object in the scene or bound to a remote theme source, like a cloud-based branding repository. Theming also makes it possible for MRTK3 to support the MRTK2 style without including an entire replicated set of UX prefabs.
- Learn more: Working with Dynamic Data and Theming in MRTK3
Get Started!
We hope you enjoy MRTK3 Public Preview and we look forward to hearing your feedback as we continue its development. To learn more, check out our comprehensive on demand videos from Mixed Reality Dev Days and join the Mixed Reality Developer program and the MRTK Holodevelopers Slack to stay up-to-date on MRTK3 and connect with the team!
Brought to you by the MRTK team: Hoff Hoffman, Max Wang, Roger Liu, Yoon Park, Finn Sinclair, Kurtis Eveleigh, David Kline, Nick Klingensmith, Grace Hsu