Control a Synthesizer with the HoloLens to Make Music in Mixed Reality

In the world of analog synthesizers, hitting a key, twisting a knob, or sliding a fader makes a beautiful musical (or not so musical) sound and can be an amazing and downright satisfying experience. Now it's about to get even more satisfying, if you add Microsoft's HoloLens into the mix as a means to twist those knobs virtually instead.

In early October, audio equipment maker Behringer showed off their HoloLens integration set up with the DeepMind 12 synthesizer at Synthfest UK in Sheffield (see second video below), the first public event to demonstrate this combination of sound and control.

For those unfamiliar with analog synthesizers like the Dave Smith, Moog, or Roland, these are devices that use a combination of sound oscillators, filters, and envelopes to create sounds from overly simple to very complex. A great, but simple, example that any proud nerd should know, is the original Doctor Who theme. A more contemporary option would be our beloved classic 8-bit consoles, which used synthesizers to produce both music and sound effects.

These days, it's pretty rare to hear music that is not layered in synth sounds. Most of these keyboards have a large number of knobs, buttons, and faders that allow the user to control and make real-time adjustments to the sounds being generated. Here is a quick visual demonstration of a synthesizer's sound-tweaking features in a normal setting.

Skip to the 33 second mark to see a good example of sound tweaking.

Out with the Old...

Behringer's DeepMind 12 is the first synthesizer they've brought to the market. It is a 12-note polyphony offering, with a fully analog signal path and a large array of control options—details that generally impress a synth player. But what's truly unique and special about it is how it was demoed at this years Synthfest UK: using Microsoft's HoloLens as an augmented reality controller for the keyboard.

Check out this demo they made:

DeepMind 12 being run through a HoloLens

With the HoloLens and what is likely Leap Motion integration, the players using their hands, manipulate virtual objects floating in space. Those objects then manipulate the sound coming from the keyboard.

These objects are tied to the keyboard through a Bluetooth connection, and use the MIDI protocol—a longtime control standard for electronic instruments —allowing the user to change the sounds on the keyboard by moving and manipulating this virtual interface. Using 3D objects in virtual space as a replacement for knobs and faders allow for far more complex combinations of changes in just a single movement. Much like many of the forward-thinking ideas coming along with augmented reality, this opens up near infinite new possibilities in real-time sound control.

Control a Synthesizer with the HoloLens to Make Music in Mixed Reality

Behringer has been letting bits and pieces leak out for the DeepMind 12 since early-2016. The DeepMind 12 has already been released for a modest price of $999.99, and can be obtained from Sweetwater, Adorama, and other retailers. We still don't know yet whether the HoloLens interface for this keyboard was simply a demonstration or if this feature was released with the unit; The specs don't include Bluetooth anywhere, but does state Wi-Fi.