iPhone’s Portrait Mode: Past, Present, and Foreground

I cannot fathom a better way to begin this essay than to quote blogger Stu Maschwitz:

“See, our eyes actually have very deep focus, but our brains and our hearts fire at ƒ/0.95.”

What camera could he have been lovingly discussing when this was said? Maybe a Sony mirrorless, or a Canon DSLR?

The year was 2016, and the camera was the iPhone 7 Plus. Though ancient by today’s standards, it pioneered a quiet revolution in one specific area of computational photography: faking a shallow depth of field normally impossible on small image sensors. Apple called this effect Portrait Mode, an extra-credit feature by its engineering team released a few months after the phone’s launch in 2016.

How does portrait mode work?

The essence of depth of field is blurring things more as they get further away from the lens’ focal point. The more light a lens lets in, the more blurry things get. Since smartphone camera sensors and lens elements are minuscule, they let very little light in. Their images have always had a distinctive flatness where both subject and background are sharply in focus simultaneously. But Apple wanted its phones to take more traditional portraits. Its camera engineers wouldn’t let the laws of physics get in the way. Mimicking depth of field usually requires an estimation of a scene’s depth at every point, and points at various depths can be blurred according to their distance from the selected focal point.

Though first conceived in 2009, Portrait mode didn’t show up until the iPhone 7 Plus was released in 2016. The primary reason for this is simple: It was the first iPhone with two cameras on its back.

If this seems unintuitive, remember that most humans see depth because of our two eyes. The distance between our eyes creates differences in the location of landmarks between each eye’s view, which our brain interprets to create depth. This effect is called stereopsis or binocular disparity.

Once a phone can compare two distanced images of the same scene, estimating the scene’s depth becomes much easier, yet still not trivial. Just as our brains have been trained for millions of years to see depth with two eyes, our phones had to be trained to do the same. Using machine learning, the phone compares the location of landmarks between the two cameras, resulting in a depth map that shows every pixel’s distance from the camera. (This is a massive oversimplification, but still accurate to the best of my knowledge.) This might be hard to visualize mentally so I’ll show a sample depth map here:

Pixels closest to the camera are bright, while those furthest away are dark, with shades of gray representing what lies between.

Some iPhones have sensors that assist in generating depth maps. My iPhone 13 Pro, for example, has a LIDAR sensor on the back. This sensor shoots out infrared light points. Where and when these points hit in the scene provides the phone with low-resolution, high-precision depth data as a sort of ground truth that helps confirm the depth map generated with binocular disparity isn’t too far off.

Many iPhones also have sensors on the front which are used to recognize face shape for biometric authentication. For selfies, Portrait mode relies more heavily on these sensors. Even the latest iPhones have only one front-facing camera so binocular disparity is out of the question.

Then how do certain single-camera iPhones such as the iPhone XR and the iPhone SE support Portrait mode? To recall a prior analogy, the human brain can still interpret some depth from just one eye’s vision— it’s just a lot more difficult. A human with only one eye can still tell if something is near or far, but they have to rely on hints other than binocular disparity. Trained algorithms can do the same. These algorithms are just good enough for Apple’s ”zero artifacts” standard, but they don’t hold up to true binocular disparity in comparison.

Depth of field is important for many reasons. The human eye has a variable f-stop, depending on pupil dilation. This means the depth of field of human vision can be very shallow. Without stereoptic vision, depth is very difficult to convey. Depth of field bokeh adds visible depth to a two-dimensional image.

How has portrait mode improved since its first iteration?

Portrait mode on the iPhone 7 Plus showed a lot of promise but its flaws were plain to see, for a number of reasons. The depth maps it generated lacked any fine detail and often assumed features like glasses or ears were part of the background, resulting in inaccurate blurring. The phone’s tiny camera sensors and more primitive image processing pipeline just couldn’t capture the world the way we see it. Even its photos taken without Portrait mode were too often noisy, overexposed, and undetailed, compared to images taken by traditional cameras.

My first iPhone with Portrait mode was the iPhone 8 Plus, which had only minor camera improvements over the iPhone 7 Plus. It’s a good enough benchmark for Portrait mode’s early stages. Below I’ll photograph the same subjects with an iPhone 8 Plus, an iPhone 13 Pro, and a Canon EOS Rebel T6 for the sake of comparison. (I’m using a 50mm prime Sigma lens on the Canon, by the way.)

Let’s break down the iterative improvements that led to the 13 Pro being near indistinguishable from the Rebel T6 in our first example. First, the image processing brings out more fine detail and widens dynamic range, preventing highlights from overexposing. The depth estimation did a better job at keeping the background blurred between small foreground objects and cutting out foliage edges. Shallower bokeh simulation (which can be manually adjusted) and iOS 16’s foreground blur are the cherries on top.

Apple’s new, aggressive approach to processing faces is laid bare here. Ensuring even exposure on the skin takes priority above all else. In many ways, the 8 Plus’ photo is more realistic with regard to lighting, due to its contrasted look, while the 13 Pro’s photo just feels more perfect than reality.

“Bokeh” refers to the shape of out-of-focus light. This shape conforms to the aperture opening of a lens, which is usually circular.

I am unqualified to explain the process of digitally faking bokeh. Such explanation requires knowledge of GPU processes, floating points, and other similar computer science buzzwords. As usual, however, I am qualified to be the arbiter of what looks good. The bokeh effect of Portrait mode has definitely improved year by year, to the point where it could likely fool a trained eye in certain scenarios.

On the iPhone 8 Plus, bokeh took on a dull appearance, with softer edges and a uniform shape across the frame. The effect was pleasing yet unconvincing. (To be clear, Portrait mode has always performed a blur effect much more complex than a simple gaussian blur since the beginning, contrary to popular belief.) The iPhone 13 Pro comes out swinging with bright and sharp bokeh, even distorting its shape near the frame’s edges in a realistic manner. Simulating bokeh in this complex manner on a smartphone, nearly in real-time, is incomprehensible to me. Once iOS 16’s public beta was unleashed upon my device, the new foreground blur left me floored. Portrait mode finally runs what I’d consider a full simulation of photographic depth of field, where it felt incomplete before.

And here’s one comparison of depth maps:

The depth map generation has improved considerably with the finer details like branches and pine needles.

How could Portrait mode improve further?

Still, I have a few major criticisms of Portrait mode remaining. It’s held back by arbitrary limitations on the distance at which you can place your focal point. Quite often I’ll back up to frame a subject in the telephoto frame, only to be told I must get closer to my subject to activate portrait mode. Before foreground blur’s implementation, an argument could be made against long-distance focal points. As the subject gets further away, bokeh blur in the background is diminished and often isn’t worth simulating. Foreground blur makes up for this on a DSLR, and could well do so on iPhones as well— if Apple allowed it. Removing this roadblock could open up so many new composition options for photographers. (I have found that you can momentarily trick the phone into placing its focal point far off into the distance, but it quickly corrects itself. I’ll include examples below of what that looks like.)

This drives me crazy since the phone is evidently more than capable of placing its artificial focal point at any distance.

Since Portrait mode is just software trickery, it can’t be combined with ProRAW shooting, so the images it produces are inflexible. A RAW file is an exact representation of the light received by each of a camera sensor’s pixels. Apple could, in theory, fabricate a new RAW file with the blur effect applied to the original RAW pixels. It’s not unholy for Apple to bake some processing into RAW files— in fact, the company already applies its own Smart HDR and Deep Fusion processing to ProRAW files in conjunction with the demosaicing step. Another option would be for Apple to let its blur be applied in photo editing apps like Lightroom, which is much less likely. Apple presumably wants to keep third-party hands away from its proprietary algorithms.

Portrait mode isn’t the default shooting mode on iPhone. It lacks the beloved Live Photo feature, skips out on some of Apple’s more intense image processing (which some would argue is a benefit), and its results aren’t ideal for all camera use cases.

However, the introduction of Cinematic mode’s video depth of field effect opens the door to the same effect on Live Photo animations, higher processor speeds could enable more intense image processing in Portrait mode, and the blur effect could be toggled on or off both pre-capture and post-capture. All of this would exponentially increase file sizes and processing power requirements, but technological improvement tends to negate these issues in due time. I’d wager that Portrait mode will one day be enabled in the camera app by default.

Conclusion

Portrait mode is a breakthrough in allowing smartphones to take a style of picture that should be impossible on anything but a large, expensive interchangeable-lens camera. Its recent inclusion of foreground blur makes the effect thrillingly beautiful and convincing.

This type of wizardry is the reason I find passion for modern consumer electronics and their advancements. It’s the reason I started this blog in the first place. When something captivates me in this way, I share it here with all of you. Portrait mode’s future will be quite something.

Thanks for reading. I’ll write again soon. 💛