DxOMark Camera Sensor

So how fair is the DxOMark Camera Sensor Score?

There is a difficult, but important question. Probably every image quality scientist[i] in the world would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws. Reassuringly, results like Figure 1 also appear to be pretty consistent with traditional hands-on reviews: camera models that were stronger [or weaker] than state-of-the-art at the time when they were introduced (such as the Canon 40D [or 50D]) show up as expected in Figure 1. And, as mention at the start of the article, having a pretty solid metric by an independent party is better than never-ending discussions about what the ultimate benchmark might look like.

The list of critical notes, suggestions and open issues are relatively subtle because the entire topic is a bit subtle:

Low ISO bias

If you compare the DxOMark data in Figure 7 for a number of prominent cameras you would get a more balanced impression about which camera to buy than by just looking at the overall DxOMark Sensor score. If you focus on the latter, you would strongly prefer the Nikon D800 with its excellent low ISO dynamic range. But this emphasizes one aspect of the sensor (essentially the ability to do single shot HDR) that provides a capability we never had in the past. It is a feature which we may infrequently need – and one that some types of users may never see (e.g. if you shoot JPG).

However, at sufficiently high ISO, other models win. High ISO usage may be a more relevant usage for many users than HDR ability at low ISO.

One can therefore ask whether DxOMark hasn’t overstressed low ISO noise[ii]. This may explain why some reviewers arrive at different conclusions about the image quality of the Canon 5D3 (or 1Dx) compared to the Nikon D800 (or D4)..

To DxOMark’s credit, the user does get three detailed scores to choose from. So you can focus on “dynamic range” if you need single-shot HDR like capability and “low light ISO” if you need to boost your ISO settings often.

Comparing different sensor sizes

As pointed out by Falk Lumo[iii], the fact that larger sensors tend to have higher DxOMark scores than smaller sensors is not a guarantee that bigger is better – even if you check out the actual DxOMark Sensor scores before selecting a camera.

Say you are considering an APS-C model like the Fujifilm X-E1 versus a full frame camera. Falk Lumo’s point is that the intrinsic definition of the DxOMark metric (as well as most other benchmarks) assumes that you would compare say Fujifilm’s 35mm f/1.4 lens to an “equivalent” 50mm f/1.4 lens on full frame. This reduces the depth of field on full frame while decreasing the noise. But if we had kept the depth of field[iv] constant (by picking a 50mm f/2.0 lens on full frame), the noise would have stayed the same. So one can argue that the DxOMark (and any other comparison across formats at constant ISO setting makes larger formats look good by assuming that we increase the total influx of light falling on the larger sensor by picking increasingly large diameter lenses (constant aperture).

Complexity of interpreting the numbers

Complexity is a fact-of-life in the high tech industry. To DxO's credit, DxOMark allows you use just a single overall score to compare camera body image quality. They alternatively allow you to zoom in (and get 3 numbers instead of one) or zoom in all the way (for graphs with the actual measured data). But despite or possibly due to all this data, it is difficult to translate a conclusion of “A is 20 points better than B” into what exactly you would expect to observe in actual photos. Because I initially had trouble translating the numbers into “What type of images would this difference show up in?”, I added some photos to this essay now I believe that I more or less figured it out.

The undocumented Master Formula

DxO does not document how the final DxOMark Camera Sensor score is computed from the individual Dynamic Range, Color Sensitivity and Low-Light ISO scores. I feel it should be provided as the overall score gets a lot of attention. With the formula, DxO countered, a manufacturer could attempt to optimize the overall score. But I still don’t see benefits to leaving this formula undocumented: if DxO believes the master formula is a reasonable approximation of what photographers are looking for in a camera, they should document it with a note that it is a compromise between completeness and ease of use.

Here is my own attempt at a recipe to compute the overall score from the three subscores. Start off with the DxOMark Camera Sensor score for the Leica M8 as an arbitrary reference point. This give you 58 points. Next add 4.3 points for every extra bit of Color Depth that a camera has compared to the M8. Then add 3.4 points for every extra unit of Dynamic Range that the camera has compared to the M8. And finally add 4.4 points for every factor 2 improvement in ISO relative to the M8. The formula version[v] seems to predict accurately to within 1 or 2 points.

Use case names

The Landscape/Sport/Portrait terms can easily confuse people who take this literally. I am tempted to interpret the 3 metrics as Dynamic Range (as DxO does), Luminance Noise (instead of Low-Light), and Chroma Noise (instead of Color Sensitivity). Those are quantities you find more often in reviews.

Print versus Screen mode

To compare DxOMark Camera Sensor scores between cameras with different resolutions, you need to look at the “Print” results. The overall DxOMark Camera Sensor score is “Print” level only, which is fine. For the next level of detail a viewer gets to choose between Print and Screen. This is less fine: Screen is not normally useful for end users (it can be useful for debugging your own calculations). The lowest level of data is presented in “Screen” mode only, but is not labeled as such. I would prefer to see all data to be labeled Print/Screen or –better yet– Normal/100%. Normal would stress that this is what matters. And 100% is similar to pixel peeping: here you look at the noise at the 100% crop level and loose the overview of what it means at the image level.

Why measure Color Depth at low ISO?

High-ISO chroma noise seems more relevant for photographers than low-ISO chroma noise. I doubt people are actually able to see color noise at such low ISO: it's hard enough to spot regular noise at low ISO, and chroma noise is even more elusive. I suspect that the choice to use low-ISO Color Depth is an artifact of originally trying to define a metric that matched studio portrait conditions. But I am not convinced that a studio portrait photographer typically has problems with visible chroma noise in the first place.

Metric measureable per ISO setting

It might have been simpler to have a single "perceived image quality" metric that could be measured at different ISO levels. This is particularly relevant because some cameras excel in high ISO conditions (requires a low noise floor) while others excel in low ISO conditions (requires physically large sensor). Showing a high-level graph with a single figure of merit per ISO setting might have simplified interpreting the results.

Sensor size visualization

DxOMark’s online graphs allow you to plot scores with MPixels along the horizontal axis. It would be nice to have an extra setting to show sensor size instead of MPixels. This would (just like many of the graphs in this article) cluster comparable products together. Representing sensor size as color would also help because photographers tend to consider different sensor sizes (unlike MPixel ratings) as different product categories.

Summary

• DxOMark Camera Sensor is all about noise.
  DxOMark Camera Sensor measures the noise-related image quality of camera bodies. Resolution and lens sharpness are covered by another DxOMark benchmark.
• Resolution-normalized
  In order to compare DxOMark Camera Sensor measurements, you need to look at the “Print” mode (aka resolution-normalized) results. These can be directly compared across cameras with different resolutions. The overall DxOMark Sensor score is “Print” level. In the 3 more detailed scores you can choose between “Print” (default) and “Screen”. Just ignore the “Screen” setting.
• Sensor size matters
  Increasing the physical sensor size should reduce noise. Assuming you can continue to work at the same apertures, an N× increase in sensor area should roughly give you an N× higher ISO setting assuming your new lens is just as fast as your old one. The area increase from “Four Thirds” to full-frame is 4×. See Figure 6.
• Large sensors need larger lenses
  If you choose a N× larger sensor area to get lower noise, and you can’t increase your exposure time, you will need a lens with an N× larger area to collect extra light (catch more “photons”). This allows your aperture value to stay the same. And this translates to extra weight, space, and costs. If you leave the lens diameter the same, you will get a smaller f-stop and this will essentially cancel out your larger sensor’s benefit if you shoot at a higher ISO to keep your shutter speeds the same.
• Sensor improvements enable more smaller cameras
  New sensors often outperform older sensors. A new state-of-the-art sensor may allow you to migrate to a smaller camera (for convenience) at the same image quality at the same ISO/f-stop settings. See Figure 2.
  Even without relying on innovation in sensor technology, if you can increase the crop factor by “c”, you will decrease the focal length by c, you should try to decrease the aperture setting by “c” (to achieve the same depth of field) and decrease the ISO setting by "c" (to get the noise down, while utilizing the extra light coming though your faster last). If you manage all this, you will stay on Falk Lumo's “equivalence” curve. This should produce identical looking images: same field of view, same motion blur, same diffraction, same depth of field and especially same image quality - but using a smaller sensor. Remember that this shinking trick will even work without having to use a fundamentally better sensor technology.
• Resolution doesn’t increase noise
  Increasing resolution (MPixels) for a given sensor size has no direct impact on image noise. In fact, some of the lowest noise cameras (Nikon D800, Nikon D3x, Pentax K-5, Nikon D7000, Sony Alpha 580) have relatively high resolutions. See Figure 1a.
• Winners and losers
  Various brands (notably Canon and medium-format suppliers) are running behind in terms of noise and dynamic range. Until recently, Forth Thirds suppliers were also running behind (Olympus fixed this in their recent models). Particularly Nikon and various newer or formerly less prominent camera brands that use Sony sensors are taking the lead in terms of image noise and particularly dynamic range. As this comparison is strongly influenced by each brand’s latest models, this situation can conceivably change rapidly (as it did when Nikon overtook Canon in terms of noise performance back in 2007). See Figure 2.
• Definitions
  You can’t directly compare DxOMark’s measurements to numbers from other sources. Each source has a slightly different definition or measurement approach. Example: DxOMark Dynamic Range results are higher than most other sources because of DxOMark’s definition. With a bit of math skills you may be able to convert back and forth using the formulas in this article. But in any case keep in mind that there is no agreed standard across benchmarks. Example: other sources typically don’t normalize noise to a fixed resolution.
• Small sensors are doing surprisingly well
  Very small sensors performed worse their bigger counterparts. But some of the newer small sensors perform surprisingly well if you compensate for their area handicap. These models actually outperform all other cameras per unit area.
• Largest sensors disappoint
  At the other end of the scale, the larger-than-full-frame sensors don’t perform as well as they should given their head start over full-frame and APS-C sensors. This suggests there is significant room for improvement there. See Figure 6.

About Peter van den Hamer

Peter van den Hamer is a physicist by training who has been working in the Netherlands as a scientist/architect in various large high-tech and electronics companies for 25 years. Apart from merely writing about technical aspects of photography, he also does some actual photography and has exhibited work at local art galleries. The most recent exhibition at the local town hall ended in the loss of these displayed signed and numbered prints when the building went up in flames in a bizarre[vi] incident.

Although I have a science and semiconductor industry background, I am not an image quality expert.

Both Dynamic Range and Color Sensitivity are measured at low ISO. This possibly gives a 2:1 bias towards the low ISO side and stresses differences there that may not be normally visible. As illustrated in Figure 7, top notch low ISO performance is no guarantee for top notch high ISO performance.

For details and references, see sections on Falk Lumo’s Equivalence Theorem above.

There are more aspects than Depth of Field, but this is the easiest one.

Expressed in a linear formula: DxOMark_Sensor_Score = 59 + 4.3*(ColorDepth-21.1) + 3.4*(DynamicRange-11.3) + 4.4*log2(ISO/663) -0.2. The 3 middle terms can either add or subtract points, depending on whether the camera did better or worse than the Leica M8. Expansion of the formula gets rid of the choice of the Leica M8 as a reference. Camera scores predicted by this formula differ from the published DxOMark Sensor scores by a standard deviation of 0.7. The formula tells us that the 3 subbenchmarks have roughly equal importance. And that a factor of 2 improvement in each subbenchmark would increase the overall score by 12.1 points. My guess is that the actual formula is non-linear and may use (under some conditions) coefficients of 5/5/5 rather than 4.3/3.4/4.4.

http://www.dutchnews.nl/news/archives/2012/07/town_hall_destroyed_by_fire_af.php.