Caltech Researchers Use HoloLens to Give Directions & Object Identification to the Visually Impaired

When computers have vision but people don't, why not have the former help the latter? That's the gist behind the Cognitive Augmented Reality Assistant (CARA), a new HoloLens app developed by the California Institute of Technology.

Assisted by the depth sensing capabilities of the HoloLens, the software uses a computer vision algorithm to scan the environment and identify physical objects. The app then announces the presence of objects to vision-impaired users, using spatial audio to orient users to the relative position of objects in their space.

Caltech Researchers Use HoloLens to Give Directions & Object Identification to the Visually Impaired — *Image by Caltech/YouTube*

"Imagine you are in a world where all the objects around you have voices and can speak to you," said Dr. Markus Meister, the Anne P. and Benjamin F. Biaggini professor of biological sciences and executive officer for neurobiology at Caltech in a statement. "Wherever you point your gaze, the different objects you focus on are activated and speak their names to you. Could you imagine getting around in such a world, performing some of the many tasks that we normally use our visual system for? That is what we have done here — given voices to objects."

Don't Miss: US Army Declares Microsoft the Winner in Battle for $480M AR Headset Contract

CARA, which was developed in Meister's lab by a team of scientists led by graduate student Yang Liu, offers several modes to assist users. The system's Spotlight mode reports identified objects based on the user's gaze, while Scan mode dictates items from left to right as the HoloLens scans the environment.

The team tested the application by designing a route through the Beckman Behavioral Biology building at Caltech for volunteers to follow. Each test subject navigated the maze successfully while using the HoloLens-based system. In addition, the team designed a virtual reality environment as a standardized test for other researchers to evaluate their own assistive devices.

The app is still in development, with current work including refinement and enhancement of the computer vision algorithm. The team envisions banks, hotels, and retail centers eventually deploying the software to help customers navigate their surroundings.

In the meantime, Meister and Liu, along with co-author Noelle Stiles, a post-doctorate scholar from the University of Southern California, detailed the team's findings in a research paper titled "Augmented Reality Powers a Cognitive Assistant for the Blind."

Last year, game developer Javier Davalos created a similar proof of concept with the HoloLens. His version also uses environmental scanning to identify surfaces and spatial audio to give users cues to their relative position, but instead of dictating the identified objects, the demo app emits an alert that intensifies in volume as users approach obstacles.

Increasingly, companies engaged in AR and visually assistive tools are applying the technology in a variety of ways to assist the disabled. Recently, a team of students at New York University developed an app that can translate sign language in real time, and even Magic Leap has patented similar technology. Similarly, the Starks AR headset can provide subtitles via a head-mounted display for the hearing impaired.

Observers often describe augmented reality as giving "superpowers" to mere mortals, which, if you know your superhero lore, can be a good or bad thing. So now, in terms of bestowing the gift of virtual sight to those without those abilities, at least we're using those powers for good.