Not a lot of people know this. But every time you snap a photo on your phone against a bright sky without losing your subject to shadow, or capture a sunset without blowing out the highlights, you're benefiting from a breakthrough that happened in a Columbia University lab in the late 1990s.
The man behind this revolution? Professor Shree Nayar, whose invention of single-shot High Dynamic Range (HDR) imaging now powers more than a billion smartphone cameras worldwide.
For photographers and videographers, the problem Nayar solved is intimately familiar. Since the beginnings of photography, capturing scenes with both bright and dark areas has meant choosing which parts of your image would be properly exposed and which would be sacrificed.
Shoot for the highlights, and your shadows turn to impenetrable black. Expose for the shadows, and your bright areas blow out to featureless white. Our human eyes handle this effortlessly, but our cameras would struggle... until Nayar's breakthrough.
As discussed in a recent article on Columbia University's website, Nayar worked with Sony researcher Tomoo Mitsunaga to develop an elegant solution: an image sensor using "assorted pixels".
How it works
Unlike traditional sensors where adjacent pixels are exposed identically, Nayar's design varies the exposure across neighboring pixels. When any pixel becomes over or underexposed, an algorithm examines its neighbors to determine the true color and brightness of that point in the scene. The result is a single image rich in detail across the entire tonal range.
The genius lies in the "single-shot" aspect. Earlier HDR techniques captured multiple exposures in rapid succession, then merged them into a composite. But time passes between frames (a bird flies, someone blinks, a car moves through the scene), creating ghosting artifacts and motion blur that plagued any photographer familiar with HDR bracketing. Nayar's technology captures all the exposure information simultaneously, eliminating these issues entirely.
Sony recognized the potential and commercialized the technology, integrating it into their image-sensing chips. Today, these sensors are in your iPhone, your Google Pixel, and countless other devices. Security cameras rely on them. Tablets use them. The technology has become so ubiquitous that most of us don't realize we're benefiting from computational imaging. We simply expect our phones to "just work" in challenging lighting conditions.
Philosophy of photography
For Nayar, a pioneer in computational imaging, this represents a broader philosophy. Rather than merely capturing what hits the sensor, computational cameras optically code the image, then decode it algorithmically to produce something richer and more detailed than traditional optical systems could achieve. His lab has extended this approach to 360-degree omnidirectional cameras and depth-sensing 3D cameras; technologies now essential in robotics, factory automation, special effects, and AR/VR applications.
The assorted pixel concept continues to evolve. Nayar's team has generalized the approach to create sensors that capture richer color information and even determine material properties; whether a surface is plastic, metal or fabric. These advances, he expects, will reach consumer products in coming years, further expanding what our cameras can reveal about the world.
So the next time you effortlessly capture a backlit subject or a high-contrast scene with your smartphone, remember: you're holding the product of fundamental research that reimagined how cameras could see. Professor Shree Nayar didn't just improve phone cameras: he transformed them into tools that, in some ways, see better than we do.
Check out our guide to the best camera phones
