The Mixed Reality team has recently released a major update to the stable version of the Mixed Reality Toolkit, labeled 2017.2.1.4, which includes many important features. The one I would like to show in this blog post in
, which allows to recreate in your project the same type of interaction that you can see in the Mixed Reality Portal. If you have ever tried the Cliff House experience, you would have noticed that you are able to use your motion controllers to easily move, rotate and resize any object, thanks to a set of gestures that mimic the real world. For example, if you want to resize an object you just need to press the trigger on both motion controllers and then move your arms out (to make the object bigger) or in (to make the object smaller). Also on HoloLens, after the last RS4 update (which is currently in preview and you can install
by following these instructions
), you are able to leverage the same gesture just using your bare hands.
Thanks to the last version of the Mixed Reality Toolkit, now you’ll be able to easily implement the same manipulation in your project. Let’s see how with a step-by-step tutorial.
Create your Unity project
As first thing, create a new Unity project. The suggested version for developing Mixed Reality projects is
, even if in my case I’m using the LTS branch 2017.4.1f1 which works just fine. Then download the latest release of the Mixed Reality Toolkit from the
. The version that contains the two-hand manipulation feature is labeled 2017.2.1.4. To import the toolkit, right click on
Import Package –> Custom Package
and pick the
file you have just downloaded. After that the content will be uncompressed, you will be asked if you want to import everything or just a subset of the toolkit. Make sure that everything is selected by pressing
and then press
After that the Mixed Reality Toolkit has been imported, you can setup the scene and the project to support Mixed Reality. The toolkit helps you with this by extending Unity and adding a new menu called
Mixed Reality Toolkit
at the top. Click on it and choose first
Configure –> Apply Mixed Reality Project Settings
For our needs, it’s ok to leave all the default options turned on, as in the following image:
and wait for the operation to finish. This task will make sure to configure the project to support Mixed Reality development, by setting UWP as build target, enabling the XR APIs (the set of Unity APIs and features to build AR/VR projects) and leveraging Direct3D as graphics pipeline.
Now return again to the
Mixed Reality Toolkit
menu and, this time, choose
Apply Mixed Reality Scene Settings
Also in this case, leave all the settings turned on and choose
. This task will focus on the configuration of the scene and it will automatically add all the prefabs from the toolkit which are required to build a Mixed Reality project. For example, it will replace the standard Unity camera with one optimized for HoloLens and immersive headsets.
At the end of the process, you should see your scene configured like this in the Hierarchy panel:
Adding two-hand manipulation
Now that we have created a basic Mixed Reality scene, we can add some content and implement the manipulation. You can use any kind of prefab but, since I didn’t want to use the typical boring sphere or cube
, I’ve downloaded a free asset from the Store provided directly by Unity:
It’s called Space Robot Kyle and it’s a 3D model of a robot. You can either search for it directly in the Unity’s Asset Store or download it from the web store and then manually import it.
Either ways, in your
panel you will find a new folder called
. The model that can be dragged inside the scene can be found inside the
folder. Just drag it into the scene and position it in a way that can be easily seen by the user, based on the camera’s position (which is 0, 0, 0). In my sample project, I’ve placed it at position 0, –1, 5, with a rotation on the Y axis of 180° (so that the user is front facing compared to the robot).
By default, the robot doesn’t come with a collider. We need one, otherwise the manipulation events aren’t able to affect the model in the scene. Choose the
element in the Hierarchy panel, press
and add a
. Since the robot has a peculiar shape, by default the collider won’t fill all the space used by the model. As such, press the
button and, by using the mouse, make sure that the whole robot fits inside the collider, as in the below image:
Now that you have a collider, you can add the two-hand manipulation. Thanks to the toolkit, the operation is incredibly easy. Just make sure that the
object is selected and press again the
button. This time, we’re going to add a script called
Two Hand Manipulatable.
In the panel you can configure a few important settings:
is the game object we want to manipulate. Most of the times we can leave this property empty, because we want to enable manipulation on the game object where we have attached the script (in our case, the robot model).
Bounding Box Prefab
can be used to enable or disable displaying the bounding box around the object during the manipulation. If we leave it empty, it won’t be displayed; if we want to display it, instead, we can leverage a prefab included in the Mixed Reality Toolkit. We can find it in the path
Assets –> HoloToolkit –> UX–> Prefabs –> BoundingBoxes
and it’s called
Just drag it from the
panel and drop it on this property.
can be used to set which kind of manipulations we want to enable between move, scale and rotate.
Constraint On Rotation
can be used to limit the rotation on a single axis.
One Hand Movement
can be enabled if we want to allow basic drag and drop of the object in the space using a single hand.
That’s all! You can easily try this directly in the editor: just press
and notice how, by using the simulated hands (which can be moved by pressing the SHIFT and SPACE keys on your keyboard), you’ll be able to drag and drop the robot in your environment.
Of course, you will see the best results while testing the project on a real HoloLens or on a real immersive headset. In this case, you’ll be able to experiment more easily the two-hand gesture and, other than moving the robot around, you’ll be able to resize and rotate it, based on the
you have set in the script.
If you want to test the project on a real device just choose
File –> Build Settings
Add Open Scenes
to add the current scene and press
. Thanks to the configuration we did in the beginning, all the right options should already be selected.
At the end of the build process, you’ll get a Visual Studio project that you’ll need to compile using the
Release – x86
configuration and deploy using:
option if you’re going to test the experience on an immersive headset
option if you’re going to test the experience on a HoloLens (connected through Wi-Fi or directly to the PC via USB)
In this post we’ve seen how easy it is to implement two-hand manipulation in a Mixed Reality project thanks to the latest additions to the Mixed Reality Toolkit. You can find the sample used for this blog post on
. You will need
to open it. Alternatively, you can find a testing scene also in the official Mixed Reality toolkit samples. To try it, you will need to download the Examples package from
, import it into a Unity project and open the scene in the path
Assets –> HoloToolkit-Examples –> Input –> Scenes –> TwoHandManipulationTest.unity