DxOMark Camera Sensor

Friday, January 11, 2013

Camera Article
Intro | DxOMark Score | Portrait, Landscape and Sport | How fair is DxOMark ?

A noise benchmark of 187 digital cameras by Peter van den Hamer

DxOMark Camera Sensor is a raw benchmark for camera bodies by DxO Labs. The benchmark is “raw” because it measures image quality using Raw output files. It is also raw as DxO’s data can be used to cook up camera reviews that cover more aspects than image quality.

This essay is a successor to a previous article published on Luminous Landscape in early 2011. This update explains industry trends using over 50 new camera models, discusses the impact of sensor size and provides sample photos to illustrate some of the geekier stuff.

If you only want to compare a few specific cameras, the original data on DxOMark’s website should be perfectly adequate – although I still suggest browsing all the pictures in this article. However, if you want a deeper understanding of what the DxOMark scores really mean, if you care about tradeoffs in camera design or are wondering about major industry trends like “mirrorless” and small high quality cameras, this article might be of some use.

This article hopes to bridge the gap between scientific publications about camera sensor design (which are quite inaccessible for photographers) and consumer-oriented camera reviews. I have tried to maintain some degree of readability by including lots of diagrams, by mentioning examples, by moving details to endnotes and by adding some actual sample photos.

Why create an update?

In the 2 years since completing the previous article, the number of cameras tested by DxO has increased from 130 to 187. The changes are more substantial than the numbers suggest: few of the original 130 models are still in production.

Incidentally, the oldest tested camera on DxOMark.com is Canon’s EOS 1Ds (Photokina 2002). Consequently the DxOMark data happens to span 10 years of digital camera history[i].

Camera trends

The camera industry landscape is currently undergoing a next great migration: just like the migration from analog to digital, an increasing number of users are starting to upgrade to cameras with smaller sensors because this results in smaller and thus more convenient camera sizes.

In other words, you can get “yesterday’s” full-frame sensor quality using a modern APS-C size sensor[ii]. Alternatively, you can get yesterday’s APS-C sensor performance using the best available CX format sensor[iii]. In fact, you can probably match or even exceed today’s entry-level medium-format image quality with a modern full-frame sensor (e.g. the Nikon D800E).

This is not just the standard story of electronics getting a bit cheaper or better every year. It is largely due to a jump in sensor performance in the past 2 or 3 years (largely thanks to Sony sensors). It also coincides with competition on the camera market from mobile phones. This causes traditional camera makers to focus on image quality: unless your camera is a lot better than a multi-purpose smartphone, how can you convince smartphone users to carry around an extra camera when their smartphone is always at hand?

Nowadays smartphones are adequate as everyday cameras, and integrate well with popular online social services like Flickr and Facebook. Smartphone sales are thus eating into the compact camera market, prompting camera manufacturers to introduce premium compact cameras (e.g. the mirrorless Sony NEX series) that clearly outperform smartphones. These premium compacts provide near-SLR image quality while looking less intimidating. And they actually fit into coat pockets. New DSLR models, in response, increasingly target professionals and those with significant investments in DSLR lenses. As a next step in this migration, the low-end medium format cameras are under attack by high-end SLRs (e.g. the 36 MPixel Nikon D800E) because these are more versatile and have comparable image quality at lower cost.

So this migration is driven by consumers who buy the most convenient (=smallest) camera that meets their needs. While others who in the past might have upgraded to a physically larger camera format may now choose to stick to the same sensor size. This story about sensor size and sensor quality is the central question of this article: how does sensor size impact image quality?

Tableau de la troupe

Figure 1 shows the top-level ratings for 187 cameras that have been tested by DxO Labs at the time of writing. In Figure 1, each camera is characterized with just a single number. DxOMark actually provides a total of 4 levels of info, and this single score is just the top of the information pyramid:

  1. A single DxOMark Camera Sensor score
  2. Three separate benchmarks for dynamic range, high ISO and color noise
  3. A bunch of graphs showing noise-related phenomena at different ISO settings.
  4. Multi-dimensional measurements behind tabs labeled “Full SNR” or “Full CS”.

I will mainly focus on the top two levels but occasionally reference the two other levels.

To save space, cameras in all graphs are labeled with nicknames such as “1Ds2” rather than “Canon EOS 1Ds Mark II” (1994). None of their friends call them by their full name anyway.

The data is shown for majority of point-and-shoot cameras or camera phones. This is because the DxOMark test procedure requires that a camera can generate Raw output files - such as Canon’s CR2 or Nikon’s NEF formats. This is because conversion to JPG introduces various artifacts that would influence the results, thereby obscuring the underlying differences between the cameras. DxOMark’s approach implicitly limits the scope to cameras targeted at relatively serious users. This is acceptable because typically JPG shooters are unlikely to worry about squeezing the maximum image quality out of their camera anyway.

Figure 1 shows the overall DxOMark Camera Sensor score for each measured camera. This is the number summarizing DxOMark’s measurement data into a single figure of merit. High scores indicate a mix of low noise and high dynamic range. The two concepts are closely related, but not quite identical. A difference of 15 points in the DxOMark Camera Sensor score is a worthwhile improvement: it corresponds to a “stop”, an “EV” or a factor of 2 difference in ISO settings. Differences of 5 points (or less) are hardly visible, but can be measured in a lab.

The horizontal axis in Figure 1 shows when each camera was announced. Newer models are shown in all graphs using larger dots than older models. This makes newer models stand out better in some of the graphs.

If you cannot find a major recent camera model, it means that it hadn’t been tested when I finalized this article. DxOMark gets flack on some forums for not having tests available shortly after models are in the shops. This is probably partly related to their business model, but I also suspect that manufacturers are more helpful at providing test cameras which they believe will get high scores (the manufacturers can help by supplying a production model for testing, and probably have the privilege to check benchmark results for errors before publication).

Figure 1. DxOMark scores for digital camera models launched in the past 10 years.
The colors indicate the size of the image sensor.

As shown in the legend, the colors of the dots represent the physical size of the image sensor. Orange is for the smallest (28 mm2) sensors as used in the Fujifilm X-10 or Canon S-110. So-called Four Thirds sensors (225 mm2) and APS-C sensors (330-370 mm2) are shown in varying hues of green. Full-frame sensors (864 mm2) are shown in blue, while so-called mediumformat sensors are shown in colors ranging from purple via magenta to red (2200 mm2). The color scale consists of a total of 64 color steps, so intermediate sensor sizes actually map to intermediate colors.

Current sensors span a range of roughly 10´ in terms of linear size and 100´ in terms of area. As we shall see, sensor size in cameras plays an important role in camera performance. Analogously, a 100´ range in combustion engine displacement takes you from a noisy little lawn mower engine (25 cm3) to a deafening 2.4 liter Formula 1 engine[iv].

Trends in the DxOMark Sensor score

Figure 1 already illustrates three key trends:

  1. Larger sensors typically score better than smaller sensors. This is why orange dots score low (e.g. Fujifilm’s X-10), green ones score in the middle ground (numerous APS-C models), and blue/purple/red dots tend to be at the top (e.g. Phase One’s IQ-180). We will explain why later.
  2. Newer sensor designs of any particular size (= same color dot) tend to outperform older designs with the same size. This is why the color bands are tilted: new models generally outperform older models. This trend can’t continue indefinitely due to physical limitations, but the rate of innovation in the past few years was actually higher than it used to be: the marketing guys have figured out where they want to go and the engineers knew how to get them there.
  3. There are other factors influencing sensor performance. These can be subtle differences between a company’s level of technical expertise, its R&D budget, its patent portfolio or its market strategy. Thus a smaller sensor (e.g. the 1.5´ Sony NEX-7, August 2011) sometimes actually matches a larger and newer sensor (e.g. the newer Canon 5D Mark III, March 2012) – thus masking both previous trends. In general these are temporary anomalies because, in the end, the laws of physics and marketing will prevail.

The lines shown in Figure 1 connect some models to their direct successors. Example: one line connects the Canon EOS 10D, via the 20D, 30D, 40D and 50D to the 60D.

Note that Nikon (dot-dashed lines) has in the past years overtaken Canon (solid lines) for many sensor sizes as far as fundamental sensor performance is concerned[v]. Some industry analysts are expecting Canon to catch up when it gets its new 0.18 µm sensor chip fabrication line operational[vi].

If you take the time to study Figure 1 in a Sherlock Holmsian pensive mood[vii], you will discover that the resolution of a sensor is not shown. Resolution is often interpreted as an image quality indicator by the general public, but has no direct impact on the DxOMark Camera Sensor score. DxOMark Camera Sensor benchmarks the aspects of image quality that are complementary to resolution. We will come back to this.

DxOMark Camera Sensor scope and purpose

DxOMark Camera Sensor ratings essentially measure image noise and dynamic range. Despite the benchmark’s name, it covers more than just the sensor. It covers the imaging pipeline starting at the point where the light enters the camera body up to the recorded Raw file[viii].

Benchmark data such as DxOMark Camera Sensor help people decide what to buy or whether to upgrade. But major benchmarks also indirectly influence industry direction. This is analogous to automotive mileage or crash tests: even if no single test is perfect, vendors will try to optimize their designs to score well on major tests.

Although DxO Labs[ix] is a commercial organization, it provides this benchmark data for free because DxO measures this data anyway for their DxO Optics Pro raw converter. And providing the data helps DxO get visibility in the photography market. DxOMark’s measurement data and graphs are incidentally not in the public domain, but can be redistributed under certain conditions[x].

Noise versus Resolution

Another reason why the term “Sensor” in the name of the benchmark can be a bit confusing is that the benchmark only covers the noise performance of the camera sensor. In reality, perceived image quality is a mix of two factors that are both impacted by the sensor:

  1. image sharpness: is there enough resolution and contrast to render required details?
  2. image noise: does noise obscure details under dimly lit conditions or in dark shadows?

Engineers like to keep both separate because they are different phenomena within their formulas: in an analogy with an audio system, image sharpness corresponds to bandwidth (which frequencies can be reproduced) and noise to the hiss in an audio system.

Image noise is primarily determined by the camera body and its sensor. Image sharpness is nowadays mainly determined by the quality of lenses because it is relatively easy to make sensors with enough resolution to match the lens’ performance.

The DxOMark Camera Sensor benchmark covers only the image noise part, but measures this under varying lighting conditions and in various manifestations (dynamic range, luminance noise, color noise). Other camera properties such as the previously mentioned sharpness/resolution, but also factors like ease-of-use, robustness, frame rate and price are all out of scope.

Note that DxO Labs also publishes a second free benchmark called DxOMark Camera Lens[xi] which tests camera/lens combinations. This metric mainly[xii] covers the resolution resulting from the lens, the sensor and any other optical component. So to get a full picture of the image quality of a camera/lens combination, you can use both DxO benchmarks[xiii].

In this article we ignore the impact of sensor resolution on image quality. DxO itself suggests that you first decide how many MPixels you need for your purposes (e.g. Will you only view images on screen? How much will you crop? Will you make extremely large prints?). Any camera with enough resolution should have comparable sharpness and can then be compared using just the DxOMark Camera Sensor score. After all, there is no direct benefit in having more resolution than you will use. But as we will see, there are no fundamental drawbacks either to having surplus resolution – despite a popular misconception that high resolution results in extra noise.

Another way to look at the relevance of resolution: nowadays, unless you use expensive lenses on a relatively cheap camera, cameras tend to have enough resolution to handle what the lenses can project onto the sensor. And for most uses, 12-18 MPixels is more than enough anyway. So a properly designed noise benchmark can be used to predict image quality as long as you keep an eye on whether you have enough resolution for your needs.

Why rehash DxOMark’s data?

The data shown here is taken (with permission) from DxOMark's website. I created new graphs to stress specific trends. My graphs don’t replace DxOMark's graphs and tables: their interactive graphs are better for comparing individual camera models.

This article addresses various interrelated questions:

  • How to interpret the DxOMark Camera Sensor results?
  • How reliable are the benchmark scores and what are their limitations?
  • Why do large sensors seem to outperform smaller ones?
  • Is it really impossible to get identical results with a smaller sensor?
  • What can we learn about the camera industry from all this data?

During the journey I will slip in a basic course on ImageSensor Performance for Dummies. This is good for your nerd rating because it is actually rooted in quantum physics and discrete-event statistics. But all this will be explained without the use of formulas. Instead I will use a familiar analogy that is remarkably similar: measuring rainfall by placing measuring cups in the rain. I threw in a few Greek λetteρs to remind you that we are on the no man's land between science, engineering and marketing.

If all this gets to be a bit too much for your purposes, just concentrate on the graphs with actual benchmark data. Questions like “At what ISO setting do you expect a full-frame camera to produce prints with the same amount of noise as a Four Thirds camera at 100 ISO, assuming both are used at f/2.8?” will not be asked during the exam[xiv].


[i]

 The first unwieldy digital camera incidentally was built by Kodak in 1975 and Canon’s revolutionarily priced D30 dates back to 2000.

[ii]

 Canon’s full-frame 5D Mark II was state-of-the-art when it was launched in 2008. In early 2012, it was overtaken in both resolution and noise performance by Nikon’s entry-level DLSR (D3200) with a 1.5´ crop sensor.

[iii]

 This is the Sony RX100, although Sony doesn’t use the Nikon’s term “CX” sensor.

[iv]

 Starting in 2014, Formula 1 is planning to switch to a somewhat more environmentally friendly engine design with a 1.6 liter turbo engine (http://en.wikipedia.org/wiki/Formula_One_engines).

[v]

 The Nikon D800 and D600 have higher DxOMark scores than Canon’s competing 5D Mark III. The Nikon D3200 and D5200 outperform respectively the Canon 650D and (older) 60D. Nikon’s flagship D4 outperforms (somewhat) Canon’s equivalent, the 1Dx. Note that at high ISO, the recent Canon models are less behind than the top-level DxOMark scores suggest. This can be seen by comparing the DxOMark dynamic range scores at high ISO settings.

[vi]

See http://www.chipworks.com/blog/technologyblog/2012/10/24/full-frame-dslr-cameras-canon-stays-the-course/. The essence is that Sony (and thus Nikon) has been using analog-to-digital converters per column for quite a while. It requires a modern chip manufacturing process to fit thousands of column ADCs onto a full-frame chip. Sony also stresses its use of patented digital- rather than analog compensation for dark frame subtraction (at the pixel level). This subtraction computes the signal level per pixel relative to the empty/reset level and is called Correlated Double Sampling (in both its analog and digital form). It is unclear whether Sony’s “elegant” digital CDS technique (http://www.google.com/patents/US7375672) substantially contributes to the excellent performance of current Sony sensors. Sony suggests that this is the case.

[vii]

http://en.wikipedia.org/wiki/Silver_Blaze

[viii]

 This means the benchmark covers the impact of semi-transparent mirrors, anti-aliasing filters, color filter arrays, the sensor itself, signal amplifiers, analog-to-digital convertors, and subsequent signal processing done inside the camera. Arguably, the decoding of the Raw file is also in scope, but this should have no impact.

[ix]

http://www.dxo.com

[x]

http://www.dxomark.com/index.php/en../About/Company/DxOMark-Conditions-of-Use

[xi]

http://www.dxomark.com/index.php/Lenses/Camera-Lens-Ratings

[xii]

 The DxOMark Lens score emphasizes resolution, but unlike typical resolution benchmarks, it provides a bonus to wide aperture lenses: these lenses may not be be optimally sharp at full aperture, but by capturing more light at full aperture, they effectively reduce the amount of noise in low light situations.

[xiii]

 Given that perceived print quality is a mix of noise and resolution, it may sound tempting to ask that both factors are merged into a single score. DxOMark Sensor score don’t do this. This is presumably because there is no real objective answer possible to the question whether camera/lens combination X or Y produces better image quality.  X may have less noise (which is sometime a problem) while Y may scale better to larger print sizes.

[xiv]

 The answer is “400 ISO”. The short version is that the Four Thirds sensor has ¼ of the surface area as a full-frame sensor. So it has the same noise level as a full-frame sensor which is underexposed by 4´. Note that Falk Lumo would likely disagree with the question. Instead he would ask, at what aperture and ISO value would a Four Thirds camera produce indistinguishable results to the full-frame / 100 ISO / f/2.8 image. See Table 1.