How the iPhone 8 Camera Works: Photo, Video, and Augmented Reality

For many smartphone users, the camera is just as, if not more important than the phone itself. Apple knows that, devoting a big section of Tuesday’s unveiling of the iPhone 8 and iPhone 8 Plus to detailing all the incredible new ways people will be able to take photos, record videos, and play around with augmented reality.

“It has an all new 12-megapixel sensor,” Phil Schiller, Apple’s senior vice president of worldwide marketing, said of the camera during the presentation. “It’s larger and faster. It provides 83 percent more light, and it’s more power-efficient at the same time. It has deeper pixels and a new color filter. This adds up to having better color saturation, a wider dynamic range of color, and lower noise in your photos and videos.”

The iPhone 8 Plus offers a dual camera system, with a wide photo lens with f/1.8 aperture and a telephoto lens with f/2.8 aperture. This combination is behind the biggest innovation of Apple’s latest camera, which uses both cameras and some nifty AI to change the lighting of backgrounds in portrait mode.

“When you compose a photo in the camera app in portrait mode, the dual cameras sense the scene,” said Schiller. “They create a depth map, they separate the portrait from the background, and you see with machine learning it actually recognizes facial landmarks.” This gives the camera the ability to isolate the individual from the background, offering an ability to alter the lighting that would otherwise only be possible if a professional photographer were composing the scene.

“These aren’t filters,” Schiller added. “This is real-time analysis of the light on your subject’s face.”

Portrait mode lighting is in beta, but Schiller said it will ship immediately with the iPhone 8 Plus.

The video camera offers similarly fine-grained control, as it performs realtime motion and video analysis to optimize the camera’s video encoder algorithm.

The iPhone 8 divides each frame into two million individual tiles, running separate analyses of each tile every second to look for details like textures and edges to figure out if that tile has grass or sky or movement in it.

Augmented reality is the final — and increasingly important — feature the camera allows for. Schiller said the iPhone 8’s cameras are individually calibrated in the factory to be optimized for augmented reality, complete with all-new gyroscopes and accelerometers to gather the precise sensory data needed.

Here the iPhone’s new A11 Bionic chip is crucial to the overall effect. Its central processing unit handles world tracking, the graphics processing unit renders the realistic graphics, and the image signal processor handles realtime lighting. With the hardware and software fully integrated to support augmented reality, the iPhone 8 figures to push this still evolving tech to its limits.