HoloLens applications have the power to overlay digital objects on top of the real world. Getting these holograms to look visually accurate comes with its own set of challenges, but how should they sound? This post will cover the audio side of developing for mixed reality, providing some tips and tricks for making your application’s audio more immersive.
Adding audio to your application is an easy way to increase user engagement and confidence. For example, even the simple act of adding a sound to confirm a button click can let the user know when their input has been recognized. Going one step further – enabling spatial audio for that same button click can make the user interface feel like it is coexisting in the same physical room as the user, rather than just some button floating in space. This short video demonstrates these differences (best listened to over headphones for the closest representation of what a user will hear with HoloLens).
To get started, you’ll need to add an audio spatializer plugin to your Unity project. I recommend including the Microsoft Spatializer. This has the added benefit of offloading all spatial audio processing to dedicated hardware on the HoloLens 2, freeing up more CPU for your application’s core business logic. If you’re looking to get started quickly, you can download this plugin directly from the releases tab on Github. If you need to customize the plugin for any reason, you can also build it directly from the repository – it is fully open source.
Once you have the plugin added to your Unity project, you can enable spatial audio for an audio source by clicking the Spatialize checkbox and setting the Spatial Blend to 1.0. You’ll also notice that there is a volume roll-off curve. Unity’s default curve may or may not work for your application. For example, we recommend disabling distance attenuation for UI sounds. To do this, flatten the curve out as seen below. Don’t be afraid to experiment with this volume curve! All applications have different requirements, so different curves can be useful.
What about for sounds for more interesting holograms? Spatial audio has the benefit of helping a user distinguish between multiple sound sources playing simultaneously. Instead of a cacophony of sounds, users can enjoy increased ability to locate holograms when spatial sound is enabled, as seen in this video.
One important note about the Microsoft Spatializer – to leverage the hardware acceleration capabilities of the HoloLens 2, the Unity audio mixer functionality is bypassed. If you are accustomed to routing audio sources to different mixer groups to better control volume or effects processing, you’ll need to apply those effects on a per-source basis instead of using the audio mixer. Unity comes with some built-in audio effects, such as EQ filters and reverb. Additionally, some other Unity plugins, such as WebRTC, rely on the audio effect infrastructure to operate properly. If you want to use audio effects in conjunction with the Microsoft Spatializer, make sure to check the “Spatialize Post Effects” checkbox on the audio source, otherwise these components won’t work properly.
A quick note on when not to use spatial audio. Given how easy it is to enable spatial audio, you may be inclined to enable it for all the audio in your application! However, there are cases when this can be distracting. In general, spatial audio works best for things that also have a visual component (i.e. menus, holograms, etc.). For audio-only parts of your application, such as narrators or background music, spatial audio may actually be more of a distraction. Consider disabling spatial audio for these sounds so the user may enjoy them without looking around for some visual cue of where they are coming from.
If you want more information about adding spatial audio to your application, definitely check out the full set of documentation, and try out the spatial audio tutorial. If you’re developing with the Unreal Engine, these same principals apply but there’s a slightly different setup process. You can learn more about using spatial audio in the Unreal Engine here. If you’ve done all this and are ready to explore what else you can do with audio, check out Project Acoustics! It can add an additional layer of immersion to your application by simulating room effects. If you’ve gotten stuck somewhere along the way, let us know in the comments, or feel free to file an issue on the Microsoft Spatializer issues page.