Transform Your Friend into an AR Musical Synthesizer via This HoloLens 2 NFT App

One of the oldest electronic musical instruments is the theremin, a synthesizer that generates sound based on hand gestures, as featured in the classic "Good Vibrations" by the Beach Boys.

Now, indie developer Lucas Rizzotto has taken the concept of the theremin and translated it to an augmented reality synthesizer app for HoloLens 2 that can turn people into musical instruments.

• Don't Miss: Control a Synthesizer with the HoloLens to Make Music in Mixed Reality

Yes, technically, people are already musical instruments, or at least those gifted with singing talent or possessing the ability to clap on beat, but bear with us here.

After teasing his project on Twitter (which we, in turn, shared last week), Rizzotto has unveiled more details on his project in a (lengthy and dramatic) video on his YouTube channel.

The app works via granular synthesis, or the deconstruction a music sample where the fragments can be manipulated to create new sounds. In his initial iteration, Rizzotto created a virtual cube filled with a particle cloud of sound fragments. Users can interact with the cube via hand gestures to generate notes.

Lucas Rizzotto/YouTube

In the final form of the instrument, Rizzotto configures it to use the arms of his partner as the target for the sound particles as he gestures over them.

"This was by far the most rewarding augmented reality project I've ever built," said Rizzotto in the video description. "I think it strikes the perfect balance of exciting, crazy but also deeply meaningful."

Lucas Rizzotto/YouTube

Unlike most prototypes, this one is available to try for yourself, if you have a HoloLens 2 or a PC-tethered VR headset and subscribe to Rizzotto's Patreon page. He also plans to release the app as an NFT.

Rizzotto's AR theremin comes on the heels of the AR Synth feature in the Google Arts & Culture app, where users can play AR instruments by interacting with them via touchscreen gestures. However, Rizzotto's prototypes not only frees users from the boundaries of the touchscreen but also demonstrates how AR can create new paradigms of creation.

Cover image via Lucas Rizzotto/YouTube