The Crazy Process Behind the HoloLens Optics That Make Mixed Reality Possible

Tricking your eyes into seeing 3D images isn't all that hard in movies or even in virtual reality, but when you start projecting holograms into the physical world, you run into some difficult problems. Microsoft obviously figured them out with their HoloLens, but how? The process is pretty amazing.

Tom's Hardware takes a deep dive into the HoloLens' inner-workings and offers a simplified but easily digestible explanation of the difference between virtual reality headsets and mixed ones like the HoloLens:

With a VR HMD, you essentially have two tiny monitors millimeters from your face, and you view them through glass lenses. By contrast, HoloLens spits out a projected image, which is then combined, diffracted, and layered to produce images you can see in space.

— Seth Colaner/Tom's Hardware

But that only scratches the surface of why this is so complicated. Let's discuss what a mixed reality headset has to contend with in order to render 3D images in your real-world environment.

The Crazy Process Behind the HoloLens Optics That Make Mixed Reality Possible — *Image via Tom's Hardware*

First, images projected on a movie screen or inside of a virtual reality headset have a fixed perceived depth. Holograms that have to stay in the exactly right place in your real-world living room need to map the depth of that space. The HoloLens accomplishes this with a depth camera and, of course, software to process the data. On top of that, it needs other sensors to track your body and eye movement so the holograms stay where you expect them—you know, like they if they actually existed.

That, alone, is a mind-bogglingly difficult problem to solve, and yet it's not even the hardest. In order for you to perceive holographic objects in physical 3D space, digital image data has to be thoroughly processed through several systems. It starts with micro displays that receive projected images. Those images then have to be combined in order to appear in three-dimensional space.

So now we have 3D objects, but they need to actually appear to exist in the real world. Combined with the necessary depth, motion, and eye tracking capabilities, Microsoft had to implement something called "defraction gratings." Made in a special glass coating, they create waveguides that essentially expand the light of the projected image to create a larger field of view. People may often complain about the limited FOV on the HoloLens, but Microsoft did quite a lot of work to even achieve as much as they did.

It's not only necessary to do all of this to provide a better field of view, but to keep holograms from "swimming" and causing symptomatic nausea. The HoloLens does a ton of work just to get crisp, clear, and comfortable holographic images to your eyes while making them look like they belong in the real world. It's not perfect, but it sure is impressive when you realize how many problems had to be solved in order to make mixed reality possible.