Header Banner
Next Reality Logo
Next Reality
HoloLens
nextreality.mark.png
Apple Snap AR Business Google Instagram | Facebook NFT HoloLens Magic Leap Hands-On Smartphone AR The Future of AR Next Reality 30 AR Glossary ARKit Dev 101 What Is AR? Mixed Reality HoloLens Dev 101 Augmented Reality Hololens How-Tos HoloLens v. Magic Leap v. Meta 2 VR v. AR v. MR
Home
HoloLens

Caltech Researchers Use HoloLens to Give Directions & Object Identification to the Visually Impaired

Man wearing smart glasses in a modern, softly lit corridor.

When computers have vision but people don't, why not have the former help the latter? That's the gist behind the Cognitive Augmented Reality Assistant (CARA), a new HoloLens app developed by the California Institute of Technology.

Assisted by the depth sensing capabilities of the HoloLens, the software uses a computer vision algorithm to scan the environment and identify physical objects. The app then announces the presence of objects to vision-impaired users, using spatial audio to orient users to the relative position of objects in their space.

"Imagine you are in a world where all the objects around you have voices and can speak to you," said Dr. Markus Meister, the Anne P. and Benjamin F. Biaggini professor of biological sciences and executive officer for neurobiology at Caltech in a statement. "Wherever you point your gaze, the different objects you focus on are activated and speak their names to you. Could you imagine getting around in such a world, performing some of the many tasks that we normally use our visual system for? That is what we have done here — given voices to objects."

CARA, which was developed in Meister's lab by a team of scientists led by graduate student Yang Liu, offers several modes to assist users. The system's Spotlight mode reports identified objects based on the user's gaze, while Scan mode dictates items from left to right as the HoloLens scans the environment.

The team tested the application by designing a route through the Beckman Behavioral Biology building at Caltech for volunteers to follow. Each test subject navigated the maze successfully while using the HoloLens-based system. In addition, the team designed a virtual reality environment as a standardized test for other researchers to evaluate their own assistive devices.

The app is still in development, with current work including refinement and enhancement of the computer vision algorithm. The team envisions banks, hotels, and retail centers eventually deploying the software to help customers navigate their surroundings.

In the meantime, Meister and Liu, along with co-author Noelle Stiles, a post-doctorate scholar from the University of Southern California, detailed the team's findings in a research paper titled "Augmented Reality Powers a Cognitive Assistant for the Blind."

Last year, game developer Javier Davalos created a similar proof of concept with the HoloLens. His version also uses environmental scanning to identify surfaces and spatial audio to give users cues to their relative position, but instead of dictating the identified objects, the demo app emits an alert that intensifies in volume as users approach obstacles.

Increasingly, companies engaged in AR and visually assistive tools are applying the technology in a variety of ways to assist the disabled. Recently, a team of students at New York University developed an app that can translate sign language in real time, and even Magic Leap has patented similar technology. Similarly, the Starks AR headset can provide subtitles via a head-mounted display for the hearing impaired.

Observers often describe augmented reality as giving "superpowers" to mere mortals, which, if you know your superhero lore, can be a good or bad thing. So now, in terms of bestowing the gift of virtual sight to those without those abilities, at least we're using those powers for good.

Cover image via Cal Tech/YouTube

Apple's iOS 26 and iPadOS 26 updates are packed with new features, and you can try them before almost everyone else. First, check Gadget Hacks' list of supported iPhone and iPad models, then follow the step-by-step guide to install the iOS/iPadOS 26 beta — no paid developer account required.

Related Articles

Comments

No Comments Exist

Be the first, drop a comment!