What’s New in DXOMARK’s Camera protocol?

At DXOMARK, the evolution of our protocols is a continuous process, aimed at keeping pace with the accelerating innovation in smartphone imaging. With each generation of devices introducing new technologies and user-centric features, our testing methodologies adapt accordingly, not only to stay up to date but also to ensure our scores remain relevant and meaningful for real-world users.

Today, we officially unveiled the 6th version of our Smartphone Camera Protocol, the most advanced and user-aligned protocol to date. This release is the product of our traditional multi-phase development strategy that reflects both technical rigor and human-centric evaluation.

A methodological framework grounded in real-world use

Each update to our protocol follows a structured methodology built on major key pillars, starting with identifying user needs and preferences, developing representative and repeatable test scenarios, and culminating in the thorough evaluation and scoring of products.

Understanding user needs and preferences

Our foundation lies in in-depth research, including multi-year investigations about user preferences through DXOMARK Insights. These studies, conducted with large panel groups, explore key user pain points, particularly in challenging domains such as HDR portrait photography. Since 2023, we have carried out extensive studies across China, India and Europe, uncovering detailed insights about user expectations in portrait photography.

To enrich this broad understanding, DXOMARK regularly collaborates with independent experts in their fields (photographers, video makers…). Ahead of the launch of our Camera v6 protocol, we are deepening our engagement through the creation of the DXOMARK Expert Committee, a body of professionals and academic experts who provide valuable perspectives on emerging trends and real-world usage scenarios.

Designing representative and repeatable test scenarios

At our state-of-the-art labs in Boulogne-Billancourt, we design tests that mirror real-world usage with scientific precision. Our approach combines objective measurements and perceptual testing, covering a wide range of lighting conditions, motion scenarios, and diverse skin tones. Purpose-built setups and proprietary tools enable us to achieve a new level of granularity in testing. Beyond the lab, we also conduct tests and analyses in varied natural environments. Each device is tested extensively with over 4000 photos captured and 200 minutes of video recorded under a wide range of conditions.

Scoring: The tip of the iceberg

While our scores are publicly visible, they represent just the surface of a comprehensive evaluation process. These scores distill the end-user experience into a clear, comparable format. With the launch of our sixth-generation protocol, we introduce a revamped scoring architecture and weighting system, aligned with an updated testing matrix and newly refined quality metrics.

What’s new in the new version of our protocol?

The sixth version of our protocol introduces updates across three key areas:

• • Enhanced HDR evaluation, featuring a new testing process and refined scoring methodology.
  • Updated portrait testing, informed by recent global studies to better reflect real-world user expectations.
  • Expanded focus on zoom performance, with particular attention to video zoom capabilities.

Now, let’s dive into the details of our protocol updates

Portrait evaluation: adapting to user trends and expectations

Taking portrait photos is one of the most common and emotionally resonant use cases in smartphone photography. Guided by our extensive insights run between 2023 and 2025 in different parts of the world, we identified 3 key elements looked at by users while defining a good portrait picture: a consistently well exposed face and overall picture, natural skin tones as well as an accurate and neutral white balance. We also observed that users were consistently unhappy and still identified challenges when it came to lowlight and nigh photography, which still represented major challenges for users.

These insights provided us with clear guidelines on users’ expectations as well as general trends on preferences, that directly drive our methodology and our tests.

What have we changed in our evaluations?

To better reflect real-world usage, including both everyday and challenging situations, we’ve significantly upgraded our portrait testing protocols:

• • 50 new portrait scenes, covering a full range of lighting conditions, from moonlight to sunlight, captured both in natural environments and simulated lab settings. In total, we now evaluate 9 lighting conditions in photo and 13 in video.
  • A broader spectrum of skin tones, ensuring inclusive and comprehensive evaluation across diverse subjects.
  • Three motion profiles, simulating real user behavior, from static handheld shooting (two-handed grip) to walking scenarios (for video), to test performance in typical portrait capture situations.

 

Our tools and test methods have evolved to deliver deeper, more meaningful insights into image quality. At the core of this is our newly developed All-in-One Portrait Lab setup, powered by Analyzer, designed to simulate real-world challenges in a controlled environment. It includes:

• • Two high-fidelity mannequins representing deep and fair skin tones, used to assess facial detail preservation.
  • Dynamic lighting simulation, covering a wide range from 0.1 lux to 10,000 lux, to evaluate performance under various illumination levels.
  • Motion simulation tools, including moving objects, a hexapod (six-axis motion platform), and a time box to rigorously test autofocus accuracy and motion blur.
  • Reflective and transmissive gray scales, supporting in-depth analysis of noise and contrast behavior.

To complete the evaluation of portraits, we now run a systematic perceptual evaluation of flare, which can affect facial clarity and background rendition.

HDR: A consistent evaluation of the HDR formats

As outlined in our recent publications, HDR is reshaping the landscape of smartphone photography. As brands explore various approaches to HDR integration (see our China Insights), new creative opportunities are emerging, alongside fresh technical challenges. The evolution of HDR formats now includes standardized versions that are compatible across a wide range of smartphones, ensuring more consistent user experiences.

To reflect the growing importance of HDR, we have now integrated a dedicated and systematic evaluation of HDR performance into our testing protocol, applicable when the tested device supports a publicly documented format that is compatible with common HDR viewing tools. This enhancement ensures a more accurate and comprehensive understanding of how HDR impacts image quality across devices.

It includes:

• • Expanded HDR scenes coverage: our testing now includes a broader range of natural scenes (across all lighting conditions, including night scenes) as well as in controlled lab environments using our AF-HDR setup.
  • New lab-based metrics: Additional objective measurements offer finer granularity in assessing HDR performance under reproducible conditions.
  • Perceptual analysis with professional reference HDR display: Evaluations are conducted using our AQuA tool (which brings an objective perspective on perceptual analysis) on an ISO 22028-5 reference HDR monitor.

When a supported HDR format is detected, images are processed with the appropriate gain maps and evaluated through our HDR visualization pipeline using dedicated scoring criteria. If the image is in a non-HDR format or an unsupported HDR format, it is analyzed as an SDR image using the same tools, ensuring consistency and fairness across all devices.

Zoom: An increased focus on growing user features

Zoom capabilities have emerged as a key differentiator among flagship smartphones. Increasingly valued by users, zoom is now widely used across a variety of scenarios, from close and mid-range portraits to long-range landscape and wildlife photography. In recent years, we’ve observed significant advancements across devices, enabling users to capture high-quality images even in the most demanding conditions.

In response to evolving user behavior, we have redefined our zoom testing protocol with a stronger emphasis on emerging use cases, such as video zoom, which is increasingly used during live events and concerts to capture subjects from a distance.

Key evolutions in our testing include:

• • A focus on the 50–150mm zoom range, which is especially relevant for close- and medium-range portrait photography.
  • Simulation of user motion
  • Evaluation criteria covering a broad set of attributes: from static elements like face exposure, contrast, dynamic range, and texture, to temporal aspects such as stabilization and autofocus consistency, as well as usability metrics like zoom smoothness.

While we’ve refined our protocol for close- to medium-range zoom, representing most of everyday use cases, ultra zoom (200 mm and beyond) continues to be evaluated through a dedicated protocol. We’ll soon publish updated results from this specialized testing.

Video: Simulating Real-Life Movement and Light

Smartphone video performance has advanced significantly in the past few years with results now getting closer to professional standards. Videos are now marked with significantly richer color, enhanced contrast and greater detail. For the past eight years, devices like the Apple iPhone have consistently set the benchmark for mobile video quality, delivering reliable performance and excellent detail retention across a wide range of lighting conditions.

To stay aligned with the rapid advancements in smartphone videography, we have significantly updated our evaluation protocols. In the sixth version of our video testing protocol, we’ve introduced several key enhancements:

• • Simulated user motion in the lab: We are the first to incorporate a protocol that evaluates video quality using captures recorded under controlled, simulated user movement bringing greater realism and reproducibility to our tests.
  • Broader range of use cases: We’ve expanded scene diversity to include a wider variety of skin tones, better reflecting real-world usage.
  • Extended lighting scenarios: Our automated lab setup now covers four distinct lighting levels (from 5 to 1000 lux), each paired with systematic HDR scene simulations. Additionally, we’ve implemented a dedicated night-shooting plan, designed to evaluate performance across a variety of low-light situations and user scenarios.

Revised Architecture and Scoring System

In the latest version of our protocol, we have revised the scoring methodology to provide a more detailed and user-relevant evaluation of device performance. The updated framework now includes two main sub-scores: Photo and Video, each assessing the performance of the device’s primary focal lengths: main, tele, and ultra-wide.

This structure offers a clearer view of how each focal length performs in both still and motion capture. Additionally, we’ve introduced use-case scores to reflect real-world scenarios, providing insights into the device’s capabilities in specific contexts such as portrait photography, zoom performance (across both photo and video), and low-light shooting—a persistently challenging condition identified in our previous research.

Initial Results from Camera v6 protocol

With the new protocol version comes an updated camera ranking, resulting in some shifts in smartphone positions compared to previous rankings.

To give you a clearer idea of what to expect, this section presents an overview of the evaluation of three popular devices.

Apple iPhone 16 Pro Max

In our Camera v6 protocol, the iPhone 16 Pro Max is mostly impacted in our Photo score. Indeed, while fine noise has a reduced impact compared to earlier versions, faces frequently appear underexposed. This lower brightness is generally less appreciated by users, resulting in a greater negative impact on perceived exposure quality.

Highlights on the performance of the product evaluated under the v6 Camera protocol:

• • Portrait: in our new protocol, the iPhone 16 Pro Max remains an excellent choice for portrait pictures, whether capturing a single person or a group on the same focal plane. Thanks to effective HDR management, portrait images appear immersive, vibrant and visually appealing.
  • Zoom Video: Video continues to be a strong area for the iPhone 16 Pro Max under our new testing protocol. When analyzing zoom performance during video recording, the device delivers smooth transitions and maintains high image quality throughout the zoom range.
  • Lowlight: With the inclusion of more low-light and challenging scenes in our protocol, the iPhone 16 Pro Max continues to perform strongly in photo mode while remaining the top performer in video. It produces bright images with a wide dynamic range, preserving both detail and contrast even in difficult lighting conditions.

Xiaomi 15 Ultra

Under our Camera v6 protocol, the Xiaomi 15 Ultra benefits from the increased emphasis on portrait color and telephoto zoom performance, resulting in a higher ranking.

Highlights on the performance of the product evaluated under the v6 Camera protocol:

• • Portrait: in our new protocol, the Xiaomi 15 Ultra was capable of capture nice portraits with realist skin tones and good exposure across all lighting conditions.
  • Zoom photo & video: The Xiaomi 15 Ultra delivers a strong performance in telephoto, keeping a high level of detail and sharpness across the entire zoom range. It also performs strongly in video zoom, offering stable and clear results.
  • Lowlight: The device provides good low-light imaging experience, featuring a warm white balance that preserves the ambient atmosphere, along with impressive noise reduction.

Samsung Galaxy S25 Ultra

In our new protocol, the Samsung Galaxy S25 Ultra is mostly impacted by the lower impact on noise and the growing weight of telephoto zoom.

Highlights of the performance of the product evaluated under the v6 Camera protocol:

• • Portrait: The Samsung Galaxy S25 Ultra delivers strong portrait photography performance across default, bokeh, and tele modes, with good subject detail, accurate edge detection, and versatile features like realistic blur effects and adjustable lighting.
  • Zoom: The Samsung Galaxy S25 Ultra offers impressive telephoto performance with sharp, detailed images across medium to long zoom ranges, supported by fast and reliable autofocus. While it delivers strong overall quality, some softness and noise appear at extreme zoom levels, placing it just behind top competitors like the Oppo Find X8 Ultra and Xiaomi 15 Ultra.
  • Lowlight: In low-light conditions, the camera generally delivered good exposure and accurate white balance, though occasional underexposure and unnatural tones were observed. Testers also noted inconsistencies in noise and detail between shots, highlighting a lack of consistency in performance across challenging lighting conditions.

Conclusion

With the launch of DXOMARK’s sixth-generation Smartphone Camera Evaluation Protocol, we reaffirm our commitment to providing the most accurate, relevant, and user-centric assessments in the mobile imaging space. By integrating cutting-edge testing tools, global user insights, and real-world use cases into our methodology, Camera v6 marks a significant step forward in how smartphone camera performance is measured. As innovation in mobile photography accelerates, this new protocol ensures our rankings remain not only scientifically robust but also truly reflective of the everyday experiences and expectations of users worldwide.