What we Test and Score
in Camera

We evaluate the performance of the smartphone while recording videos in a large number of use cases, various lighting conditions and environments from natural low light to indoor and outdoor conditions.  

Our tests are performed:

• • Using the resolution setting and frame rate that provides the best video quality. If a main camera offers 4K video mode but uses 1080p Full HD by default, we’ll select 4K if 4K provides a better image quality (sometimes 4K is not stabilized which lead to a lower image quality).
  • In an extended range of lighting conditions from 1 lux for lowlight to 1000 lux for bright light conditions and even higher for actual outdoor scenes
  • With a tripod or through real or simulated handheld conditions from steady shot to some very challenging walking or even running scenarios.

To evaluate video, we evaluate seven quality attributes, the same as the ones applied for photo, with the addition of stabilization:

Important note:

Our range of acceptable renderings for all attributes in various light conditions are based on the findings gathered from our multi-cultural focus groups. We believe that within this inclusive scope, the risk of bias is eliminated when evaluating a smartphone camera for a particular attribute. In keeping with this approach to testing, we do not evaluate manufacturers’ signature renderings.

We evaluate the performance of the camera while zooming in and zooming out for still images and videos, from ultra-wide to very long-range zoom.

The Zoom evaluation breaks down into two key parts:

We evaluate all image quality attributes, including artifacts. Every attribute is key in the experience, and we do a homogenous image quality evaluation for the complete zoom range. Some criteria may be more challenging than others. For example:

• • For Wide: distortion (straight lines appearing curved in an image), face deformation or chromatic aberration, level of detail and level of noise in the field of view 
  • For Tele: Texture preservation at various distances whether the device keeps a high level of details at different zoom distances 

Tests are performed for more than 10 zoom distances, for photo and video. In video, the smoothness of the zoom process during video recording is also evaluated.  

The camera app preview has a huge impact on user experience and gives you a first impression of image quality. A photographer might simply choose not to capture an image at all if the preview looks like it’s not worth pressing the shutter button, even though the recorded image would have been absolutely fine.

We evaluate the consistency between the image seen on the screen before the shutter is pressed and the final image rendering.

To evaluate preview, we particularly check:

• • Default mode: Every image quality attribute is evaluated with a particular attention to overall exposure and HDR capabilities.
  • Bokeh and Portrait mode: Simulation of the bokeh effect is properly applied to the preview
  • Zooming photo: zoom smoothness of the exposure, color and focus adaptation when zooming from the minimal to the maximal zoom factor available

This sub-score covers the popular yet non default mode often called Portrait or Bokeh in modern smartphones. We evaluate the overall image quality with a simulated depth of field effect. As smartphone cameras do not provide a natural lens effect, they need to simulate the depth of field.

We check how well the camera separates the subject from the background when taking portraits without accidental blur on the focus plane, as well as the quality of the gradation and naturalness of background blur, which makes the subject stand out. In addition, we evaluate the traditional portrait image quality attributes (such as exposure, color, texture, etc.)

Tests are performed with the bokeh dedicated mode within the camera application