Introducing the new DxOMark Mobile test protocol

When we launched our original DxOMark Mobile test protocol in 2012, among the first smartphone cameras we tested were the Apple iPhone 4 and 5, the Samsung Galaxy S2 and S3, the Nokia 808, and the HTC 8X. Back in the day, those were all high-end devices with very capable cameras.

Apple’s iPhone 4 was the first smartphone we tested on DxOMark Mobile. Its 5 Megapixel camera featured a 1/3.2-inch sensor with a 1.75 µm pixel size and a f/2.8 aperture.

A lot has happened in the five years since then. Smartphone photography has pretty much eradicated the compact camera market segment, and the mobile sector has become the number one driver of innovation in imaging. The latest smartphone models come with camera technology and features that have slowly but surely brought our original test protocol to its limits.

One of the last devices that we tested using the old DxOMark protocol, the HTC U11, combines a large 12 Megapixel 1/2.55-inch sensor with a fast  f/1.75 aperture, optical image stabilization and phase detection AF.

  • Advanced CMOS image sensor technology and more powerful chipsets allow for real-time multi-frame processing, resulting in substantial improvements in low-light performance.
  • On-sensor phase detection and laser-based time-of-flight sensors have drastically improved autofocus speed and precision, especially in low-light conditions.
  • Dual-camera setups have brought optical zooming capabilities and bokeh-like features that simulate on smartphone cameras the shallow depth-of-field of a fast lens on a DSLR. Other dual-cams combine the image data from an RGB sensor with the information captured by a secondary monochrome chip for improved detail, reduced noise levels, and increased dynamic range.

Apple’s iPhone 7 Plus was one of the first smartphones to offer optical zoom in a dual-cam setup:

To take these developments into account and to be prepared for the future, we have redesigned the DxOMark Mobile test protocol to include new and expanded outdoor and lab test scenes and evaluation methodologies. With the new protocol, we capture and analyze more than 1500 images and two hours of video for each device. Compared to the previous protocol, it provides the following new elements:

  • A new zoom sub-score that is based on extensive testing at multiple focal lengths
  • A new bokeh sub-score based on lab and outdoor testing
  • Low-light testing down to 1 Lux
  • Motion-based test scenes for more accurate evaluation of camera performance and processing techniques in real-world situations

Many smartphone components, such as displays and chipsets, have become commoditized in recent years, but camera quality is more than ever a key differentiator for manufacturers and a major decision criterion for smartphone buyers. The new DxOMark Mobile test protocol enables consumers to make informed buying decisions by evaluating the true camera performance of the latest mobile devices and by challenging OEM marketing claims.

You can find a detailed overview of the new DxOMark Mobile test protocol here. Click here for a comparison of the original and new protocols, looking at the scores of some of the best performing smartphone cameras. The smartphone reviews that followed the previous DxOMark Mobile protocol are still valid for evaluation purposes, but please keep in mind that you cannot compare their test results with those obtained with the new protocol.

  • Pingback: Apple iPhone 7: Retested with new DxOMark Mobile protocol - DxOMark by DxO - Image Quality()

  • Kumbari Di Schiffer

    waiting for samsung galaxy note8 and LG V30

    • GCHQ

      Note 8 is good and the LG V30= extremely oversharpened pictures

    • Never Trust Hillary

      Keep waiting, Apple paid them to ignore Note 8.

    • I agree with you.

  • Feder Grand

    waiting for HTC 10

    • GCHQ

      HTC U11 is the best

  • TinusVerdino
  • LifelessNewt

    Waiting for the top “Budget” OnePlus5, Xiaomi Mi6 and those top Leica lens from Huawei, and see how they can do against the mainstream brands.

  • Gcrack

    Waiting for Sony Xepria XZ Premium review! 😉 (at last now we know what kept you occupied that long! 😉)

    • GCHQ

      Keep waiting + 1 year

    • RockStar2005

      So XZ Premium and new iPhones are the only phones rated under NEW system. But new system is basing its scores on like who can do Bokeh and who can’t. Who the F cares about Bokeh? lol

      We’ll have to wait and see how Note 8 does, but I don’t buy how iPhone camera got better but Xperia camera got worse? Both X Perf and iPhone 7 Plus got an 88 last year. Doesn’t add up that newer phones would do substantially WORSE. What do u guys think??

    • troll

      Earn money on lending coins. It’s easy

  • Rino Lazzizzera

    please test Lg G4 or G5 with the new protocol 🙂

  • مُحمّد™

    waiting for other devices (note8 , LG V30 , s8 , etc ……)

  • troll

    S7? S8?

    • Erwin

      Yea plz test S8 and S7 🙂

  • Ebubekir Özbek

    iphone up dxqmark down, sided review

  • Joms

    Good job keeping those iCraps visible within top 10 when they should have been buried deep down by al least 15 Android phones.

    • MrWalker1000

      makes me wonder if this was done to favor apple devices to the top.

      • Daniel Toledo Fuentes

        What makes you think that?
        That the test protocol changed just a few days before Apple Event?
        That they have tested iPhone 8 and 8 Plus in record time?
        That now iPhone 8 and 8 Plus are in the top of the ranking?
        That now the protocol include Dual Camera test for “zoom and bokeh-likes effects”? (why not B/W or wide angle photos?)

        • MrWalker1000

          also bs how the sony rated so low. at least pixel is still number 1 as expected.

    • GCHQ

      Android is rubbish

  • Юра Мочернюк

    Are you going to add OnePlus 3T?

  • Erwin

    S8 please 🙂

  • T T

    So how much have Apple sponsored you to change your ratings towards zoom and bokeh? Such convenient timing.

    • MrWalker1000


      • MrWalker1000

        im starting to like them less and less now.

    • Brad Fortin

      If Apple was sponsoring them they wouldn’t have waited nearly a year to review the iPhone 7 Plus, and they wouldn’t have changed their methodology from one that ties the 7 Plus and the Pixel to one that favours the Pixel over the 7 Plus. If anything it seems like Google sponsored them to keep the Pixel at the top of the ratings for months while they casually ignoring the 7 Plus for almost a year.

      • Daniel Toledo Fuentes

        Now they took only a few days to test 8 and 8 Plus, and protocol changed just a few days before Apple Event… Now they’re both at the top of the ranking…

        • Brad Fortin

          That doesn’t change the fact that the new protocol downgraded the iPhone 7 and 7 Plus compared to the old protocol.

          If you go over any of the reviews for the iPhone 8 and 8 Plus you’ll find that just about every reviewer agrees that they have the best cameras this year. They’ve clearly made some big upgrades.

          • Daniel Toledo Fuentes

            iPhone 7 Plus did not have a test with the old protocol. For me it seems that Apple paid some $$$ this year to have all their phones tested with a very convenient new protocol.

            Everything I have heard is LG V30 being a really new and best camera. For new iPhones I just read that they are good upgrades if you are coming from a 6S Plus.

    • One of the last devices that we tested using the old DxOMark protocol, the HTC U11, combines a large 12 Megapixel 1/2.55-inch sensor with a fast f/1.75 aperture, optical image stabilization and phase detection AF.

    • troll

      Earn money on lending coins. It’s easy as 1 – 2 – 3

  • Daniel R. Pinheiro

    Ok, so now it’s time to retest the Lumia 1020, huh?

    • GCHQ


  • Abdullah Kösem

    Will you do macro test for each phone?

  • Mehr Frank

    Still waiting for the Lumia 950 camera in the lineup…

    • GCHQ


      • Mehr Frank

        I’m still undecided, whether I should throw in a dead fish or not.
        Apart from the Windows Phone is dead debate, the Lumia 950 still has one of the best cameras in the smartphone world. OIS, PureView oversampling, a real Zeiss lens instead of just Zeiss optics…
        So it’s not just me, waiting for a DXOmark rating of it…

    • troll

      Earn money on lending coins. It’s easy as 1 – 2 – 3

  • AngryTeuton

    LG G6 re-test please

  • MiKa Ka Fung

    Very nice! please test oneplus5

    • Danny Xavi Hernandez Diaz

      yes please, re-test Oneplus 5

  • Shafiq Jetha

    I would love to see the current and previous generation Windows Mobile devices added to this list. The Lumia 950 XL would probably blow away a lot of the completion, even over a year after it was discontinued.

  • Nomaned

    Congratulations on the new web design and comments system.
    it would have been better to wait until next year for the new protocol.

  • ozhan

    waiting for nokia 8 zeiss camera !

    • HDN900

      And ? Look on Nokia 808 Pure View…i wanna seee sth like this in Android / HMD Global-Nokia.

  • MrWalker1000

    new testing protocol is pretty bs. Zoom and Bokeh are really not necessary elements of the test. And given that apple has benefited in score from these new tests its making me skeptical about just how truly objective you guys are.

    • Chris Perry

      Zoom and bokeh aren’t real aspects of mobile photography, other than maybe in a test scenario

      • Not true at all. Bokeh can easily come into play when taking macro shots, and what about portrait mode (software “bokeh”).

    • Gabriel Lim

      “To take these developments into account and to be prepared for the future,…” If you don’t like progress, then LEAVE!

  • GCHQ

    iPhone 8!!!

  • Rick_Deckard

    Where is the Oneplus 5 result?

  • Never Trust Hillary

    How much did Apply pay you to fast track reviews for the iphone 8? Note 8 was released before iphone 8 and still no ratings for it.

  • Kygo Muhamadou

    Do you have other blogs about mobiles cameras test that more thrust than this.

  • Bruno Vieira

    HTC 10 and S7 New protocolo?

  • Bruno Vieira

    it is not possible that after the creation of a new protocol only the iphone 7 of a jump in the score and the HTC u11 not? Retry S7 and HTC 10 devices! At least these devices and HTC u11 tested in the new protocol should present improvements in the score also or only the apple devices that take advantage with the new protocol?

  • flodxomark

    Thank you for your suggestion. We try and test as many cameras and devices as possible with our new protocol. There are many to come, so please holding on!

  • J. Carter

    Please please PLEASE test the LG V30!!!

    • flodxomark

      Hello. Unfortunately we are not able to communicate on our testing roadmap. I invite you to stay tuned for further announcements. Thanks !

      • jean-francois Cheron

        You’re suppose to be the reference in image quality and you did not test the next big thing in Mobile Photo… so much for a reference…

        • flodxomark

          Dear Jean-Francois, Given our limited resources, our team have to prioritise some new mobile models for release but there are many more to come. Thank you for patience. Regards

  • ralf L

    Hallo why you Not Test the LG v30 in DXO mobile id IT too Bad or to good👍😀please make the Test ist so importent Thank you a Reader from Germany

  • BigEnso

    Can we still see the prior protocol’s results? Not all of us are moving up to the latest and greatest. I’m still using – and happy with – my Note 4.

  • BigEnso

    “Dual-camera setups have brought optical zooming capabilities …”

    Dual cameras with fixed focal length lenses do not produce optical zooming capabilities. They are, in effect cameras with fixed focal length lenses. Optical zoom requires one lens that goes through a range of focal lengths within the lens and not by changing of lenses. For example, my Note 4 has a fixed focal length lens of 31mm (at 35mm equivalent) and the Note 8 has 2 fixed focal length lenses of 26mm (35mm equivalent) and 52mm (35mm equivalent). Neither of these cameras has optical zoom despite what terminology the manufacturer may use. Even the Pixel 2 with its single lens does not do optical zoom but relies on digital zoom.

  • portier

    Why isn’t there any DxO test concerning Fuji cameras and lenses ?
    Thank you for your answer(s).

  • Fabio P

    If you change the way you benchmark at the very least consider to update ALL your previous toplist entries.
    Skipping S7 & Co was not only un-professional but also makes me smell there is a (money) interest in this.
    Booooooooooh from my side.

  • Emmanuel Ogunmadewa

    Please can you guys re test the Sony Xperia z3+ again with your new procedure.

  • Yantz

    Please can you test old smartphone xiaomi Mi Note with your new procedure??

We are hiring!