Portraits are the most valued and popular type of photography, yet capturing great portraits remains technically challenging for smartphone cameras. The specific issue of achieving accurate skin tones, for instance, has received considerable attention from researchers and manufacturers alike.
After taking into account all previous work on this topic, DXOMARK’s image quality experts conducted their own extensive qualitative study aiming to:
● identify trends in users’ preferences regarding portraits (pictures of a single person as well as a group of people)
● identify elements of satisfaction and frustration;
● explore the technical challenges.
To achieve this, we designed a unique methodology that allowed us to gather detailed insights from the most common use cases, environments, and conditions. This scientific methodology was used first for our European study in Paris, but it could be easily applied to other regions of the world, to other areas of study as well as other electronic products (laptops, for example).
We present it to you here.
The question of perception
Perception is a challenge when it comes to portrait quality evaluation. Indeed, people’s preferences when it comes to photos are often tied to their memories and familiarity with the subject. Hence, we hold our portraits and those of other people to different standards.
This begs the question: which qualities can most people agree a “good portrait” should have?
To find answers, DXOMARK’s image quality experts conducted this new qualitative study aiming to identify the reasons of frustration and the key pain points in smartphone portrait photography.
Understanding and measuring user preferences
The methodology our experts built allowed them to achieve two main objectives.
Understanding user preferences
This requires a comprehensive analysis that encompasses all smartphone camera uses (meaning types of portraits here), and the variety of conditions they take place in. Of course, each usage presents unique technological challenges.
Only by fully understanding each of these uses and the technical difficulties they pose could we simulate highly accurate test conditions.
Measuring user preferences
This is a critical component of this analysis. The scale should represent the perceived quality of each type of portrait according to an individual or a test group.
Conducting this analysis with a large group and creating a test scenario that closely resembles real-world usage was key to the success of this study.
Analyzing portrait preferences in relation to skin-tone rendering
We took on the question of skin tone rendering quality perception in smartphone portrait photography.
● gathering a panel of people representative of all skin tones but also diverse cultural backgrounds, age groups, and gender inclusive
● developing a relevant shooting plan
A shooting plan includes a set of diverse photos used to identify users’ portrait preferences. We designed such a shooting plan for many years. Quantitative surveys are very useful per region to understand the preferred use cases. In the context of this study, a “relevant” plan is one that covers most use cases.
The shooting plan: A key component to anchor users’ insights in real life
The technical framework
The shots had to be taken by professional photographers. Why? Because we needed perfectly comparable shots. The challenge lay in accurately capturing the same scenes with different devices.
Our goal was not to compare devices or evaluate their performance but rather to gain insight into user preferences regarding the top offerings on the market.
Therefore, we used the most advanced smartphones, the flagship devices, available at the time of the study, as well as a professional digital camera that allowed us to look at what the future of smartphone photography may hold.
⚠️ For each scene and type of portrait, four different devices were used: three smartphones and a professional camera.
These allow us to identify the main trends in preferences, each of which can then be studied in more depth.
Scenes and stages
The shooting plan was tailored to the specifics of the geographical area under study. In our case, it was Paris. Our professional photographers curated a plan that embodied the look, feel and essence of a European way of life. Our ultimate goal was to capture images that would resonate with the European panel.
💡This type of study can be replicated anywhere in the world, with local photographers capturing and showcasing their respective regions’ unique customs and traditions.
A stage refers to a combination of:
● lighting conditions
● framing (scene composition)
● number and position of respondents
A total of 180 stages were shot, with one, two, three and four models in each.
A scene is a specific combination of a stage and models placed within it. The model is the variable between different scenes within the same stage.
Our shooting plan was comprehensive to include all types of scenes. The were partitioned in the following way (the number represents the number of scenes shot for each condition) :
“We understand that certain test conditions, especially night scenes, can be more challenging than others. That’s why we included a large variety of conditions during testing.
We also enriched the shooting plan with lab scenes, which feature models in front of a white background, under neutral and consistent lighting. This controlled environment allowed us to focus only on the rendering of the portrait and its reception by the panel, without the scene, light or conditions affecting the results.”
Hervé Macudzinski, Image Science Director, DXOMARK
The light conditions
We shot a total of 1,620 pictures in HDR, SDR and backlit conditions, partitioned as shown in the following chart:
● backlit (very challenging light conditions)
● SDR or Standard Dynamic Range (limited range of brightness and colors)
● HDR or High Dynamic Range (high range of brightness and colors)
We put together a panel of European people representing all skin tones as well as a variety of cultural backgrounds, genders, and ages. A total of 123 people participated in the survey, with 83 models/respondents photographed in 405 scenes, 30 professional photographers, and 10 DXOMARK image quality experts, making this one of the largest studies of its kind.
Both genders were almost equally represented, with the panel made up of:
● 52% women
● 48% men
Every adult age group was included as well:
● 18 to 30 years old (25%)
● 30 to 40 years old (29%)
● 40 to 50 years old (19%)
● 50 to 60 years old (15%)
● Over 60 years old (12%)
To select and classify the respondents based on their skin tone, we used the Fitzpatrick scale, a tool used to determine how different skin types react to the sun. The scale organizes skin types into 6 distinct categories, all included in our study:
● Type I, “light” (12%)
● Type II, “fair” (30%)
● Type III, “medium” (23%)
● Type IV, “olive” (2 %)
● Type V, “brown” (8%)
● Type VI, “deep” (4%)
Unveiling the Satisfaction Index
The DXOMARK Satisfaction Index is a numerical representation of user preferences. It is a combination of two distinct aspects that we measured in this study: One measures preference and the other measures rejection. By combining these two results, we were able to gather insights not only about user preferences but to quantify them as well.
Participants took all the tests under controlled viewing conditions and were unaware of the devices that were used to capture the images.
The details of how we created the DXOMARK Satisfaction Index are presented below.
The two-step user survey
Step 1: The best picture
First, participants were presented with only two images, side-by-side, and asked to select the one image that they preferred based on its overall image quality.
In order to quantify the perceived difference in quality our experts used a Just Objectionable Difference (JOD) scale.
At the end of the survey, for each participant, every image taken with each of our four cameras was given a preference score. We could then aggregate those results to estimate a preference score for groups of participants.
Step 2: Social media-worthy picture
In the second part, participants were presented with four images of the same scene taken with different cameras (one with a professional camera and three with smartphones), and then asked to identify which image they would not want to post on social media, effectively, which image or images they would reject. The goal of this question was to refine our preference analysis.
Why social media?
We wanted to measure acceptability. Our question was: “what do respondents consider to be the minimum level of quality acceptable?”. In that regard, social media provides a criterion that speaks to everyone yet remains significant. If we had simply asked people which photo they would keep, they might have chosen a lower-quality option because of their sentimental attachment to it.
“We needed a criterion for evaluating the quality of photos. Social media suitability proved to be the ideal one.”
Hervé Macudzinski, Image Science Director, DXOMARK
Calculating the Satisfaction Index
After conducting this two-step survey, we collected the following information for each scene:
● the overall rejection rate for all respondents
● the rejection rate for the group being studied
● the JOD scale
With the collected data, we used the formula below to calculate the Satisfaction Index score per picture, and we scaled the result so that it would fit within a range of 0 to 100.
Taking into account the confidence interval for each portion of the index, we could also determine a confidence interval for the overall Satisfaction Index.
The Satisfaction Index falling below 70/100 meant the photo may not meet user expectations.
Conversely, examining the characteristics of photos with scores above 70 helped us identify the prevailing preferences within a given use case. This understanding allowed us to establish the commonalities between satisfying renderings, as well as their technical characteristics.
Why use a Satisfaction Index?
The Satisfaction Index is a homogeneous and comparable score that can be used to compare participants, groups of participants or scenes.
By examining the Satisfaction Index for each individual, we can gain valuable insight into their ability to identify image quality issues and their preferences compared to other participants or groups, and the trends in their preferences.
Closing thoughts and considerations
This study analyzes the impact of shooting conditions and camera choice on image quality perception, but it also answers other related questions that provide us with consumer insights:
● Are smartphone users currently satisfied with the quality of their portraits?
● Do all high-end smartphones provide the same level of satisfaction in this respect?
● Do professional photographers produce more satisfying images overall compared to smartphones?
● If so, which gaps in quality can non-photographer users perceive?
● Does age influence quality perception?
● Does gender affect quality perception?
● Does the perceived quality of pictures remain the same regardless of the model’s skin tone?
● What other dimensions influence the respondents’ choices and their perception of image quality?
The complete study by DXOMARK Insights highlights the technical parameters that are key to ensuring high-quality portraits and user satisfaction. To manufacturers, that is vital information.
Stay tuned for the first results, coming soon!