Introducing the new DxOMark Mobile test protocol

When we launched our original DxOMark Mobile test protocol in 2012, among the first smartphone cameras we tested were the Apple iPhone 4 and 5, the Samsung Galaxy S2 and S3, the Nokia 808, and the HTC 8X. Back in the day, those were all high-end devices with very capable cameras.

Apple’s iPhone 4 was the first smartphone we tested on DxOMark Mobile. Its 5 Megapixel camera featured a 1/3.2-inch sensor with a 1.75 µm pixel size and a f/2.8 aperture.

A lot has happened in the five years since then. Smartphone photography has pretty much eradicated the compact camera market segment, and the mobile sector has become the number one driver of innovation in imaging. The latest smartphone models come with camera technology and features that have slowly but surely brought our original test protocol to its limits.

One of the last devices that we tested using the old DxOMark protocol, the HTC U11, combines a large 12 Megapixel 1/2.55-inch sensor with a fast  f/1.75 aperture, optical image stabilization and phase detection AF.

  • Advanced CMOS image sensor technology and more powerful chipsets allow for real-time multi-frame processing, resulting in substantial improvements in low-light performance.
  • On-sensor phase detection and laser-based time-of-flight sensors have drastically improved autofocus speed and precision, especially in low-light conditions.
  • Dual-camera setups have brought optical zooming capabilities and bokeh-like features that simulate on smartphone cameras the shallow depth-of-field of a fast lens on a DSLR. Other dual-cams combine the image data from an RGB sensor with the information captured by a secondary monochrome chip for improved detail, reduced noise levels, and increased dynamic range.

Apple’s iPhone 7 Plus was one of the first smartphones to offer optical zoom in a dual-cam setup:

To take these developments into account and to be prepared for the future, we have redesigned the DxOMark Mobile test protocol to include new and expanded outdoor and lab test scenes and evaluation methodologies. With the new protocol, we capture and analyze more than 1500 images and two hours of video for each device. Compared to the previous protocol, it provides the following new elements:

  • A new zoom sub-score that is based on extensive testing at multiple focal lengths
  • A new bokeh sub-score based on lab and outdoor testing
  • Low-light testing down to 1 Lux
  • Motion-based test scenes for more accurate evaluation of camera performance and processing techniques in real-world situations

Many smartphone components, such as displays and chipsets, have become commoditized in recent years, but camera quality is more than ever a key differentiator for manufacturers and a major decision criterion for smartphone buyers. The new DxOMark Mobile test protocol enables consumers to make informed buying decisions by evaluating the true camera performance of the latest mobile devices and by challenging OEM marketing claims.

You can find a detailed overview of the new DxOMark Mobile test protocol here. Click here for a comparison of the original and new protocols, looking at the scores of some of the best performing smartphone cameras. The smartphone reviews that followed the previous DxOMark Mobile protocol are still valid for evaluation purposes, but please keep in mind that you cannot compare their test results with those obtained with the new protocol.

  • Pingback: Apple iPhone 7: Retested with new DxOMark Mobile protocol - DxOMark by DxO - Image Quality()

  • Kumbari Di Schiffer

    waiting for samsung galaxy note8 and LG V30

    • GCHQ

      Note 8 is good and the LG V30= extremely oversharpened pictures

    • Never Trust Hillary

      Keep waiting, Apple paid them to ignore Note 8.

  • Feder Grand

    waiting for HTC 10

    • GCHQ

      HTC U11 is the best

  • TinusVerdino
  • LifelessNewt

    Waiting for the top “Budget” OnePlus5, Xiaomi Mi6 and those top Leica lens from Huawei, and see how they can do against the mainstream brands.

  • Gcrack

    Waiting for Sony Xepria XZ Premium review! 😉 (at last now we know what kept you occupied that long! 😉)

    • GCHQ

      Keep waiting + 1 year

    • RockStar2005

      So XZ Premium and new iPhones are the only phones rated under NEW system. But new system is basing its scores on like who can do Bokeh and who can’t. Who the F cares about Bokeh? lol

      We’ll have to wait and see how Note 8 does, but I don’t buy how iPhone camera got better but Xperia camera got worse? Both X Perf and iPhone 7 Plus got an 88 last year. Doesn’t add up that newer phones would do substantially WORSE. What do u guys think??

  • Rino Lazzizzera

    please test Lg G4 or G5 with the new protocol 🙂

  • مُحمّد™

    waiting for other devices (note8 , LG V30 , s8 , etc ……)

  • troll

    S7? S8?

    • Erwin

      Yea plz test S8 and S7 🙂

  • Ebubekir Özbek

    iphone up dxqmark down, sided review

  • Joms

    Good job keeping those iCraps visible within top 10 when they should have been buried deep down by al least 15 Android phones.

    • MrWalker1000

      makes me wonder if this was done to favor apple devices to the top.

      • Daniel Toledo Fuentes

        What makes you think that?
        That the test protocol changed just a few days before Apple Event?
        That they have tested iPhone 8 and 8 Plus in record time?
        That now iPhone 8 and 8 Plus are in the top of the ranking?
        That now the protocol include Dual Camera test for “zoom and bokeh-likes effects”? (why not B/W or wide angle photos?)

    • GCHQ

      Android is rubbish

  • Юра Мочернюк

    Are you going to add OnePlus 3T?

  • Erwin

    S8 please 🙂

  • T T

    So how much have Apple sponsored you to change your ratings towards zoom and bokeh? Such convenient timing.

    • MrWalker1000

      right?

      • MrWalker1000

        im starting to like them less and less now.

    • Brad Fortin

      If Apple was sponsoring them they wouldn’t have waited nearly a year to review the iPhone 7 Plus, and they wouldn’t have changed their methodology from one that ties the 7 Plus and the Pixel to one that favours the Pixel over the 7 Plus. If anything it seems like Google sponsored them to keep the Pixel at the top of the ratings for months while they casually ignoring the 7 Plus for almost a year.

      • Daniel Toledo Fuentes

        Now they took only a few days to test 8 and 8 Plus, and protocol changed just a few days before Apple Event… Now they’re both at the top of the ranking…

        • Brad Fortin

          That doesn’t change the fact that the new protocol downgraded the iPhone 7 and 7 Plus compared to the old protocol.

          If you go over any of the reviews for the iPhone 8 and 8 Plus you’ll find that just about every reviewer agrees that they have the best cameras this year. They’ve clearly made some big upgrades.

          • Daniel Toledo Fuentes

            iPhone 7 Plus did not have a test with the old protocol. For me it seems that Apple paid some $$$ this year to have all their phones tested with a very convenient new protocol.

            Everything I have heard is LG V30 being a really new and best camera. For new iPhones I just read that they are good upgrades if you are coming from a 6S Plus.

  • Daniel R. Pinheiro

    Ok, so now it’s time to retest the Lumia 1020, huh?

    • GCHQ

      No

  • Abdullah Kösem

    Will you do macro test for each phone?

  • Mehr Frank

    Still waiting for the Lumia 950 camera in the lineup…

    • GCHQ

      Who?

      • Mehr Frank

        I’m still undecided, whether I should throw in a dead fish or not.
        Apart from the Windows Phone is dead debate, the Lumia 950 still has one of the best cameras in the smartphone world. OIS, PureView oversampling, a real Zeiss lens instead of just Zeiss optics…
        So it’s not just me, waiting for a DXOmark rating of it…

  • AngryTeuton

    LG G6 re-test please

  • MiKa Ka Fung

    Very nice! please test oneplus5

    • Danny Xavi Hernandez Diaz

      yes please, re-test Oneplus 5

  • Shafiq Jetha

    I would love to see the current and previous generation Windows Mobile devices added to this list. The Lumia 950 XL would probably blow away a lot of the completion, even over a year after it was discontinued.

  • Nomaned

    Congratulations on the new web design and comments system.
    it would have been better to wait until next year for the new protocol.
    Greetings.

  • ozhan

    waiting for nokia 8 zeiss camera !

  • MrWalker1000

    new testing protocol is pretty bs. Zoom and Bokeh are really not necessary elements of the test. And given that apple has benefited in score from these new tests its making me skeptical about just how truly objective you guys are.

    • Chris Perry

      Zoom and bokeh aren’t real aspects of mobile photography, other than maybe in a test scenario

    • Gabriel Lim

      “To take these developments into account and to be prepared for the future,…” If you don’t like progress, then LEAVE!

  • GCHQ

    iPhone 8!!!

  • Rick_Deckard

    Where is the Oneplus 5 result?

  • Never Trust Hillary

    How much did Apply pay you to fast track reviews for the iphone 8? Note 8 was released before iphone 8 and still no ratings for it.