Proof of Concept: Music Everywhere Uses HoloLens to Help Piano Students Learn Improvisation

Students from Carnegie Mellon University's Entertainment Technology Center have been working on an augmented reality system to help teach music in a project called Music Everywhere.

This application for the Microsoft HoloLens would utilize holograms to visually instruct the user and teach them how to play an instrument in the process. Music Everywhere would also have a virtual band that the wearer could play along with as well as see.

The team developing this project is made up of Seth Glickman, a producer/pianist, Fu Yen Hsiao, a UX/UI designer, and programmers Shantanu Das and Byunghwan Lee. Their concept is similar to something called HoloKeys, a HoloLens concept which overlays holographic instructions onto piano keys to help the user play songs, but is much more detailed in its approach.

Teaching piano improvisation techniques and providing a space in AR for piano students to practice creating their own phrases, melodies and rhythms became the primary instructional focus of the Music Everywhere project.

We were fortunate to have a variety of resources to help us develop the teaching method. We consulted with CMU faculty, professional jazz, blues, rock and classical performers, while leveraging team member experience and drawing inspiration from related literature by Bob Taylor, David Baker and John Mehegan among others.

Holographic computing and the MIDI protocol—a long-time standard used for electronic musical instruments—are what make the goal of streamlining the musical learning process possible. The core ideas that are being focused on are:

• Improvisation: Understanding music theory to the degree that a player can improvise a melody that fits in a certain key is a tough skill to master. It is also a skill that is generally not part of normal music curriculum, but is needed when playing with others.
• Virtual Hand: Many people on this planet are visual learners, even for things like music. Being able to watch someone or, in this case, a hologram, play or chord or a phrase before trying yourself can be a great boon.
• Virtual Band: As mentioned above, having a virtual band to practice your improvisational skills give the user an open environment to learn and hone their skills—without the need to pay a group of people to help.

Fu Yen Hsiao/YouTube

• Lesson Builder: This is a set of tools that allows the instructor to build a curriculum for a student. Personalized lesson plans that know exactly where a student is at, and where they need to go, is a great way to accelerate learning.
• Overlay System: The HoloLens can draw an overlay on top of a keyboard (and hopefully other instruments in the future), signaling to the user what to do next. Then, using MIDI-over-Bluetooth, the HoloLens will receive a message of what the user actually played. At this point, the HoloLens will determine if it was the correct note or the wrong note, if this timing was correct, or if the improvised phrase was in key.

Fu Yen Hsiao/YouTube

• Genre Variety: Using different genres and styles of music allows the user to find something they can connect with, therefore further motivates them to learn more.

According to the site for this project, it was primarily to be worked on between September and December, which likely means that this was a graduate project for CMU. I contacted Byunghwan Lee who confirmed this, but who said that the team was still working on Music Everywhere so they could submit it to conferences such as SIGGRAPH (which is coming up on July 30 in Los Angeles, CA). Hopefully, we'll also see it hit the Windows Store sometime in the near future.

How many of you out there are also musicians, or are trying to be one? Would something like this have sped your learning process along? Let us know in the comments below.

• Follow NextReality on Facebook, Twitter, and YouTube
• Follow WonderHowTo on Facebook, Twitter, Pinterest, and Google+

Cover image by Fu Yen Hsiao/YouTube