Skin Tone: The next challenge in smartphone portrait photography

The quality of photos and videos has always been a key differentiating factor for smartphone manufacturers. As more and more people become avid photographers and use their phones exclusively, not in addition to a DSLR, the challenge is meeting their rising expectations.

According to a global DXOMARK survey conducted by YouGov in January 2022, portraits of friends and family are the most popular type of pictures people take. Getting the exposure right and effectively capturing skin tone is critical to a good portrait. So, what are the technical challenges to capturing skin tone rendering in smartphone photography?

Our image quality experts have immersed themselves in this still unexplored subject with the methodology and scientific rigor that characterizes DXOMARK, in its quest to better understand users in relation to what today’s technology enables.

How did smartphones replace classic digital cameras?

Cameras first appeared on cell phones 20 years ago, and a lot has changed since then. The smartphone has now become the primary camera in almost every household. In fact, they are closing the image quality gap with digital cameras, and their quality is a major selling point for manufacturers.

Focus on the power of selfies
The introduction of the front camera made it easy for users to take pictures of themselves. And the trend has only grown – in 2023, selfie use is at an all-time high, with 93 million selfies taken every day.¹ Selfies and their ubiquity have highlighted the importance of quality and rendering. Users are used to seeing photos of themselves, whether they are taken with a rear camera or a front camera.

Smartphone users’ favorite subject: friends and family

DXOMARK conducted a survey to identify users’ behavior and attitudes regarding additional features on their smartphones:

- family, friends, and pets make up more than 50% of the subjects photographed;
- on average, only 17% of respondents’ photos are travel-related, and life/food subjects make up an average of 9%.

Kodak’s original motto in 1889 was “You press the button, we do the rest”. The automatic camera mode of today’s smartphones delivers on this promise. This feature helps cameras to produce high-quality results by recognizing both the scene or subject to be photographed and the shooting conditions.

What are the key challenges of portrait photography?

Portrait photography is an art form that hinges on several key elements, with target exposure, contrast, and skin color being at the forefront!

Perception and memory colors

Memory colors are associated with the ones of familiar objects. When it comes to photography, preferences are often closely related to memory. If people know the subject well, they will have high expectations, limiting the number of possible renderings.

Skin tone is one of the biggest challenges in capturing color in portrait photography. Everyone has a clear idea of how people they know personally should “look” in a photo, in all kinds of lighting conditions.

Photographers are the first to judge whether the shot is satisfactory. In a non-professional setting, the person or people being photographed may also give input on how the photo should look. Also, in the case of friends and family, the people on the scene are very important.

Usually, the camera user is responsible for setting the preferences. It’s important to remember that preferences may differ between those who know the subject and those who don’t. The same applies to the scene in the frame – people’s perceptions can vary.

Capturing the moment

Exposure, color, and contrast are all important in accurately capturing skin tone. With exposure being the key player:

- cameras need to capture motion, which means as much information (light) as possible in a very short time;
- the great strength of a full-frame hybrid camera is its ability to capture the maximum amount of light in a matter of milliseconds

As shown in these examples, using three of today’s leading devices, camera suggestions, and by extension, user preferences, can vary widely.

The technical challenge of skin tones

Different skin tones bring different technical challenges for each scene. Each tone can make images more or less contrasty, and more complicated to expose correctly. For instance, a group shot with HDR conditions.

In these pictures, we have several subjects in the same outdoor scene, taken with the Samsung S23.
It is interesting to note that the exposure strategy is quite different from photo to photo. Skin tones and clothing colors affect the statistics that are used to feed the camera’s algorithms and then the camera’s ability to deliver optimal images. In these examples, capturing skin tones may be more challenging because the signal is lower.

Skin tones have to be considered in the context of their environment – it’s a highly complex set of parameters, with very subtle variations.

How is it even more of a challenge for smartphones?

The sensor size of a top-of-the-line smartphone is 10 times smaller than that of a reflex camera – so the challenge is completely different.
Smartphones have to do everything automatically for the user! Also, they struggle with small sensors in low light and freezing the moment, have to avoid clipping on faces and offer dynamic range.

Balancing light and image processing

For smartphones, the goal is to capture more light spatially. One way manufacturers have done this is by designing ever-woven lenses (up to f/1.4) while keeping the size of the camera module more compact.

Image processing is another tool used to improve the performance of smartphone cameras. It allows them to capture light temporally. How? By systematically capturing several images and merging them into one image!

Smartphone photography: a potential unlocked?

One of the features that makes the photographer’s job easier is the AI used on smartphones. How? By taking over some or all of the settings. AI analyzes the scene to be photographed and applies the appropriate settings.

However, there is a trade-off between the perfect camera and the time and money available to engineers to develop it. This can lead to biased AI databases. These biases can have a direct impact on the final images.

To correct this problem, the AI needs to be trained on a comprehensive dataset. This should include:

- a variety of use cases: settings, shooting conditions, skin tones;
- a variety of annotations: feedback aggregation method and point of view;
- several examples of expected renderings: according to quality criteria and user preferences.

In short

The smartphone has to become the professional camera and the professional photographer all in one.

The issue of user satisfaction: how to address it?

Another important consideration is how satisfied users are when shooting in different conditions. The YouGov-DXOMARK survey reveals that satisfaction varies with different lighting conditions, whether outdoors, indoors, in low light, or at night.

In fact, if 78% users are satisfied with their outdoor photos taken in full light, as shown in the bar chart above, only 1 in 5 photos are rated unsatisfactory… Smartphone devices and camera technology have a long way to go to meet user expectations.

With all the challenges discussed above, this dissatisfaction could increase when it comes to portraits of family and friends.

Smartphone portrait photography: closing thoughts and considerations

All in all, portraits are the most popular and valued type of photography. But along with sports and wildlife photography, they’re also one of the most technically challenging for our smartphone cameras.

Skin tone in portrait photography has been the focus of several manufacturers and the subject of numerous scientific publications. At DXOMARK, we decided to pursue this topic because it is key to understanding users, which is at the heart of everything we do. It all starts with them and their usage. We’re helping manufacturers better understand this subject and user preferences through in-depth qualitative research.

Other questions we’ve been looking at include: How can we measure preferences accurately and understand what drives them? Could cultural differences have a potential impact? And what about the influence of age and environment? DXOMARK’s Image Quality experts have addressed these issues in a new extensive qualitative study, focused on identifying the key pain points in smartphone portrait photography. It involved:

● a shooting schedule representative of the real world: numerous common scenes and scenarios;
● a panel representing every skin tone;
● a data-based approach by using JOD (Just Objectionable Difference) ²

[1]Source: Photutorial, “How many pictures are there (2023): Statistics, trends, and forecasts” https://photutorial.com/photos-statistics/

[2]JOD is a way to determine the level of perceptual differences between two images before an objection is raised. For example, two images are said to be 1 JOD apart if 75 % of observers find that one image is better in terms of image quality than the other.