DxOMark Mobile test protocol and scores

The DxOMark Mobile test protocol takes a different approach to our sensor and lens test protocols. Rather than looking at a discrete component (sensor, lens) the Mobile protocol assesses the performance of the imaging pipeline in its entirety, including lens, sensor, camera control, and image processing. This means that still image quality evaluation in the Mobile protocol is based on the camera’s JPG output rather than on RAW image data as for sensors and lenses. The protocol also covers the test device’s performance in video mode, and scoring is calculated from both objective measurements and from perceptual evaluation of more than 50 challenging and realistic indoor and outdoor scenes by a panel of imaging experts.

Testing for most image quality criteria is undertaken with the smartphone camera in its default (auto) mode. We intervene manually only to activate the bokeh (portrait) mode, the flash, and the zoom feature.

The new protocol includes the evaluation of camera performance in a wide range of real-life situations.

We designed a dedicated studio setup for the new bokeh test.

DxOMark Mobile: What’s new for 2017?

With the mobile sector being a major driver of innovation in imaging, and smartphone camera technology rapidly evolving since the original DxOMark Mobile test protocol launched in 2012, the time has come for an update. The new version of DxOMark mobile takes into account the latest developments in mobile imaging technology and adds the following components:

  • The new zoom test analyzes image output at three subject distances and in several test situations (outdoors, indoors, and in the lab). Key criteria for judging the results are resolution and image detail, noise, color reproduction, and image artifacts.
  • A new bokeh test analyzes the camera’s ability to create a pleasing background blur while keeping the image’s main subject in focus. The test evaluates the strength, smoothness, and transition of the blur effect. It also looks at distortion on portrait subjects, artifacts, and the image processing’s depth estimation capabilities.
  • When testing the autofocus and sharpness, the updated test protocol now takes into account motion in the scene and the speed of image capture. Sharpness on moving subjects with a moving photographer is measured in a simulation setup in the lab. A new autofocus test evaluates not only accuracy but also speed and texture, and we now undertake noise tests shooting handheld as well as shooting with the device mounted on a tripod.
  • For the updated low-light analysis, we take still image samples at very low light levels between 1 and 20 Lux. For our video quality analysis, we record all low-light footage in between 5 and 20 Lux. Overall light levels for most photo tests now range from 1 lux to 1000 lux, and color temperature from 2300 to 6504 Kelvin.
  • We have expanded our DxOMark Mobile video analysis to include objective measurements in addition to perceptual evaluation for most test criteria. For many tests — for example, white balance, exposure, and noise, we now also measure how quickly and smoothly the camera adapts to changing light conditions during recording.

Perceptual analysis is undertaken by a panel of experts using image quality rulers.

A new studio test measures the camera’s speed of capture — that is, the time between pressing the shutter and image capture.

DxOMark Mobile Scores

The DxOMark Mobile Photo and Video sub-scores and the overall score are generated from a combination of objective lab measurements and perceptual analyses. The following test criteria feed into the Photo sub-score:

The Video sub-score is calculated from the following criteria sub-scores:

  • Exposure
  • Color
  • Texture and noise
  • Autofocus
  • Artifacts
  • Stabilization

Scores are presented in the format as shown below.

90
DxOMark Mobile
photo
91
DxOMark Mobile
video

For a better idea of how the updated protocol impacts device scoring in practice, please have a look at our 2012 vs. 2017 score comparisons for a range of devices (including the Google Pixel, Nokia 808 and iPhone 7 Plus) that we tested using both test protocols.

  • Viacheslav Dynin

    Hey!

    What about the total score? Can it be more than 100?

    How overall score of HTC U11 can be 90, if max score HTC achieved on autofocus is ONLY 90, and other one less..

    If mobile phone achieved from different criteria 90 (Color, texture, noise and etc), overall will be 90 too, or 150?!

    If maximum score is 100, it means that, two cameras can be better than average, for example HTC U11 (perfect) and another one, mobile X from future (better than HTC U11, for example in two times, score in differrent criteria will be 95, for exampe).. And what will be it total score?! HTC 90 and Mobile X 92, or 95?! One mobile better than another in two times, but different in score only 5 points?!

    Thanks a lot

    • Lars Rehm

      Hi there, yes, the total score can be higher than 100, we just need to wait a little longer for the cameras to get even better 🙂

  • Correct! Bokeh and zoom are the future of mobile photography! Very sad for the new scores of my HTC U11 🙁

  • Nomaned

    This smells really bad, you’re going to favor the iphone

  • Nomaned

    Zoom y bokeh ? In serius, this is to favor iphones

    • MrWalker1000

      convenient timing for the new iphones too.

  • Trugvang Ericksson

    How much did Apple shell out this time 🙁

    Bokeh ..eeh and not resolution and low iso performance… Seems DXO-mark is going down the bias hill?

    • Jsmith8899

      Bokeh has existed on android forever though…right?

  • Kambis

    Optical stabilization should also enter into the Photo score. How many people are going to shoot with tripod like in the test scenario?