The 2nd level of DxOMark scores
Let’s get back to business again. In the next sections, we examine how the overall DxOMark Camera Sensor score relates to three lower-level measurements. The DxOMark Camera Sensor score is computed using (resolution-normalized) metrics for:
This 2nd level of data is measured and provided by DxOMark on their website. In fact, the DxOMark website 4 distinct levels of information, ranging from easy to use (and thus oversimplified) to almost the raw measurements (and thus for specialists):
In the colorful graphs shown here, we stick to levels (1) and (2). But I will use levels (3) and (4) once or twice to explain strange behavior.
The 3 metrics used to compute the overall sensor ranking
These three metrics are respectively shown in Figures 5, 6 and 7. As a DxO manager, Nicolas Touchard, explained during a telephone interview:
The DxOMark Camera Sensor score is under normal conditions a weighted average of noise, dynamic range and color sensitivity information. But some non-linearities are deliberately included in the algorithm to avoid a clear weakness in one area from being masked by a strength in other areas.
It is worth noting that these three underlying measurements[i] are not entirely independent because they are all caused by sensor noise: Dynamic Range is the brightest recordable signal (at low ISO) divided by the background noise. Color Sensitivity or Color Depth indicates whether small color differences are masked by chroma noise. And Low-Light ISO tells you at what ISO settings on different cameras will give you equivalent noise levels.
Although a camera that is great on one noise metric will probably perform pretty well on another, different cameras do in fact win first place in each of the 3 categories. This confirms that we are not just seeing the same data represented in three different ways.
DxO at some point tried to link the metrics to different types of photography (“Use Cases”), but DxO is fortunately starting to deemphasize this as the mapping between the 3 measurements and the Use Cases was not always helpful. Here were the mappings – for what it’s worth:
DxO Use-case name
Enough light = low ISO
This metric assumes that you use a tripod if needed. Many other types of photos can have a large contrast: architecture, portraits, night photography, weddings. A high Dynamic Range allows you to make large exposure corrections (if somehow your metering went wrong) or to do HDR post-processing using a single raw image.
This metric assumes you need to go to higher ISO. This is relevant for many other types of photography: street, wildlife, news, weddings, night, concerts, and family. Most photographers need high-ISO settings regularly.
Enough light = low ISO
This metric assumes you have enough light but may be a fair indication of what you would get in low light. Essentially it measures choma noise in the dark parts of a low-ISO image. Portraits may be less critical as chroma noise could be filtered out (at the cost of resolution) or you may be able to boost your lighting.
So all-in-all, I recommend not taking the Landscape, Sport, and Portrait naming too seriously. At best they are nicknames, and particularly “Portrait” is the least accurate of the bunch.
We will now discuss how the 187 cameras perform on each of these three metrics.
Dynamic Range at low ISO
DxOMark's definition for their Dynamic Range metric says:
“Dynamic Range corresponds to the ratio between the highest brightness a camera can capture [..] and the lowest brightness [..] when noise is [as strong as the actual signal].”
So far, this is a pretty standard definition. It tells you how many aperture stops of light (EV = bit = factors of two) can be captured in a single exposure. It is analogous to asking how much water a bucket can hold, expressed in the smallest accurately measurable unit.
Elsewhere in the documentation we find that Dynamic Range (in so-called "Print" mode) is
“normalized to compensate for differences in sensor resolution.”
This scaling calculates what the dynamic range would be if you scale the image to a resolution of 8 MPixel. The choice to show the results for 8 MPixels is not really relevant unless you want to compare DxO scores to measurements from other sources: any other number of MPixels would add a constant offset (in EV) to the published Dynamic Range scores. So you can simply forget that number – you just compare DxOMark dynamic range numbers to each other and the difference between two cameras in EV is independent of the resolution of your camera or how large you choose to print the results.
Furthermore the Dynamic Range used in the overall benchmarking is the maximum Dynamic Range as
“measured for the lowest available ISO setting” [typically between 50 and 200 ISO].
Today’s sensor with the highest Dynamic Range score, the Nikon D800(E), spans over 14 stops at 100 ISO. The results for the Pentax K-5II(s) and Nikon D7000 are actually very close[ii] to those for the D800(E). DxOMark's Dynamic Range plot for these cameras shows that their Dynamic Range drops by almost one 1 EV each time the ISO is doubled. This resembles an ideal amplifier that amplifies the sensor’s signal and noise without adding noise of its own. That is impressive.
Low ISO Dynamic Range and Canon
At present, Canon cameras underperform on the Dynamic Range test, or more precisely on DxOMark’s Dynamic Range measurements at low ISO. At high ISO, the Canon 1Dx actually has a better dynamic range than the D800(E), and the Canon 5D3 matches the D800(E) at high ISO. This confirms that newer Canon models like the 5D3 and the 1Dx are state of the art as high-ISO cameras.
But the problem with Canon (and all tested sensors except those with the latest and greatest Sony sensors) is that the dynamic range hardly increases when you go below about 800 ISO. This is consistent with test photos that show that the 5D3 cannot match the low ISO clean shadow performance of the D800(E), the K-5(II/IIs) or the D7000.
This is illustrated in Photo 2 that shows a 750x750 pixel crop of a 21 MPixel wedding[iii] photo taken at 200 ISO on a Canon 5D Mark III. The top of the image is exposed at 100%[iv], while the bottom contains 0.05% gray[v]. This implies a dynamic range of 2000:1 or 11 EV. DxOMark’s more precise measurement gives 11.65 EV of dynamic range at a nominal setting of 200 ISO.
Note that the noise at the bottom of Photo 2 exhibits horizontal “banding”. As the camera was held in portrait mode, this is normally considered to be vertical banding or “fixed-pattern column noise”. If you cannot see this (or details in the other photos) well enough be sure you are viewing this on a calibrated monitor. And you may want to save the picture and inspect it in a photo viewing application such as Photoshop, Lightroom, Capture One or Aperture.
Sony claims[vi] that they are the first and only currently supplier able to avoid visible fixed pattern (column) noise. For this, Sony uses a combination of having on-chip analog to digital convertors (one ADC per column[vii]) and a digital trick to elegantly subtract the black or empty level of each pixel. In our rain and cups analogy, it is like emptying the cup before starting the measurement, weighing it (with any residual water inside), exposing the cup to the rain, and then weighing the cup again. Subtracting the original weight avoids measurement errors due to the “empty” weight and due to measurement offsets. In sensor engineering, this technique is known as Correlated Double Sampling is standard for both digital weighing[viii] and for light sensors, but Sony has a somewhat cooler way to do this.
Canon users frequently wonder whether Canon will catch up, particularly in terms of low ISO dynamic range – given that it has lost the lead since the Nikon D3x back in late 2008. Some industry analysts[ix] expect that Canon will catch up when they start using a new state-of-the-art 0.2 mm copper[x] sensor manufacturing process which they are currently testing. This allows Canon to switch to switch from off-chip analog to digital converters to per-column ADCs.
Final thoughts on Dynamic Range
Figure 5 shows two camera models (unrelated to the new Sony sensors) that perform unusually well given that they were respectively introduced in 2004 and 2006: the Fujifilm FinePix S3 and the S5 with Dynamic Ranges of 13.5 EV. This was achieved by combining large and small photodiodes on the same sensor. The small photodiodes can capture the highlights without overflowing, while the larger photodiodes simultaneously capture the darker parts of the image with less noise. The signals from the two sets of pixels were merged digitally into one HDR image. This technology never reached widespread introduction.
Experiment: If you want to learn more by playing with the data yourself, trying looking up (under DxOMark's tab "Full SNR") the gray level at which the signal-to-noise ratio drops to 0 dB for the 80 ISO curve[xi]. For the Pentax K-5 II this is at a gray scale value of 0.008%. The brightest representable gray shade is 100%. So the ratio is 100/0.008 = 12500:1 which gives log(12500)/log(2) = ln(12500)/ln(2) = 13.6 stops.
But we are not finished. The "Full SNR" values in that particular DxO graph are “Screen mode” meaning not resolution-normalized. So we still need to correct for the K-5 II’s resolution of 16 MPixels rather than 8 MPixels. The noise scales with the square root of this ratio, thus giving an extra 0.5 stop[xii] of Dynamic Range when scaled to 8 MPixels. The resolution normalized value for the low ISO Dynamic Range listed by DxOMark should thus be about 13.6+0.5=14.1. The actual value by DxOMark is indeed 14.1.
Apart from confirming that we kind of understand how the benchmark works, this exercise shows that a 2´ difference in resolution corresponds to a mere 0.5 EV difference[xiii] between Screen Mode and Print Mode.
Low-Light ISO score
Here is DxOMark's definition for their Low-Light ISO score:
Low-Light ISO is [..] the highest ISO setting for the camera such that
This is a rather complex definition with built-in non-linearities: you are essentially supposed to increase[xiv] the ISO value until you break any of the above three rules. Due to this definition, the outcome can be anywhere in the ISO range[xv] - not just values normally considered to be high ISO.
Low-Light ISO is again computed using a reference resolution.
The general idea behind this Low-Light ISO metric is simple: it tests which ISO level still gives acceptable image quality using a somewhat arbitrary criterion for exactly what “acceptable” means. As Figure 6 shows, the best camera on this particular benchmark is the 12 MPixel Nikon D3s.
Currently the 16 highest ranking cameras on the Low-Light ISO benchmark are all full-frame sensors. So, full-frame is clearly a good choice if you regularly work in low light conditions.
The blue scaling line in Figure 6 shows how other sensor sizes would score if they would perform as well as the Nikon D3s – but corrected with a handicap to compensate for their smaller sensor size. In other words, the blue line shows what would happen if you take the same technology of the D3s sensor and make it smaller by trimming off- or simply ignoring the edge pixels[xvi].
The blue line in Figure 6 shows that the best 1.5x APS-C cameras, and the best CX-sized sensors are all on par with this hypothetical scaled down D3s. This is clearly not the case for medium-format sensors: these should be able to deliver "acceptable" (according to DxO’s definition) images at 6400 ISO. But actual medium format sensors perform 5-10 times worse than they should. Commercially this may not be a big deal because these SUVs of the camera world are often used on tripods or in studios with flashes. So, although there may not be a sufficient market for this, record breaking high ISO and high dynamic range cameras should be possible with large sensors.
Another surprise is that the smallest sensors manage to outperform the blue D3s scaling line. The orange scaling line shows that the diminutive Pentax Q is currently best at high-ISO if you assign a handicap for sensor size. This doesn't mean that the Pentax Q has very low noise. On the contrary: it needs to be operated at 200 ISO to get the same print quality as the D3s at 3200 ISO. But in view of the size handicap, various of the smaller cameras do a surprisingly good job[xvii].
Experiment: If you want to play with the ISO numbers, you can look up (under "Full SNR" for the Pentax K-2 II) the ISO setting at which 18% gray gives a 30 dB (5 EV) signal-to-noise ratio[xviii].
To get the more relevant resolution-normalized ISO value, you have to use a value of 27 dB instead of 30 dB. Those 3 dB compensate for the 2x higher resolution of the K-2 II’s 16 MPixel sensor.
Interpolation of the values for 800 and 1600 ISO gives a Low-Light ISO value of 1,353. Because the actual sensor sensitivities of cameras deviate[xix] from the nominal ISO values, we can improve accuracy by using the calibration curve measured by DxOMark for the K-2 II. This gives a result of 1277 ISO, which is 0.1 stop higher than the actual DxOMark value of 1235 ISO.
Interpreting the Low-Light ISO benchmark
Photo 3 was taken at the very end of the same wedding party, again a Canon 5D3, but at 12800 ISO. You can tell how little light there was: the man’s face in the background is partly lit by the display of a cell phone that he is apparently checking.
Because the ISO settings have been boosted by 6 stops compared to Photo 2, the noise is clearly visible at this magnification. I could have suppressed the noise by filtering or using a different raw convertor. But the point of the photo is that the banding that was visible in Photo 2 is not visible at high ISO. This is because at 12800 ISO the analog signal gets amplified 64´ compared to the 200 ISO in Photo 2. This amplifies both signal and noise, thereby masking fixed pattern noise or “banding” (as seen at the bottom of Photo 2) as introduced downstream in the circuitry.
The two photos thus show two distinct noise-related phenomena:
The two phenomena are different in the same sense that an athlete may be better at running long distances while another is better at short distances. All distances involve running fast, but athletes seldom excel at both the sprint and the marathon.
In principle, I think that a camera can be both good at high DR (at low ISO) as well as having low noise at high ISO. Figure 7 shows the DxOMark data for the top two Canon cameras and the most similar Nikon models.
You can find DxOMark’s Dynamic Range score by following each line to the left until it stops. The Low-Light ISO values for these models are around 3000 ISO, so can be compared by checking who is ahead in middle of the graph.
Again, the question is what we see from this graph:
Low-ISO Color Sensitivity
In a color sensor the green, red and blue color components are measured independently. The ratio between these values determines the perceived color. So independent noise in each channel impacts the ratios and thus impact the apparent color.
Here is DxOMark's definition for their Color Depth score:
Color Depth is the maximum achievable color sensitivity, expressed in bits. It indicates the number of different colors that the sensor is able to distinguish given its noise.
The metric thus looks at local color variations caused by this noise. It does not represent color accuracy – although a form of color accuracy data can be found deeply buried in the DxOMark Camera Sensor data[xx].
The benchmark values for Color Depth are again normalized with respect to sensor resolution. And, again, the phrase "maximum achievable" actually means that the Color Sensitivity is measured at the lowest (e.g. 100) available ISO settings.
As shown in Figure 8, larger sensors clearly tend to have a larger Color Depth score. This is largely explainable by their lower noise at full well capacity (see Figure 4). But color noise also depends on the choice and performance of the microscopic color filters that allow the photodiodes to measure color information (not shown in Figure 4). If less saturated color filters (e.g. "pink instead of red") were used, then the three color channels would respond only marginally differently to different colors. This would lead to a higher general/luminance sensitivity of the camera, but would introduce more color noise.
For more information on the role of the "color response" of color filter arrays, see this white paper[xxi] where DxO points out the impact of differences in color filter design between the Nikon D5000 and the Canon 500D[xxii].
A Color Depth value of 24 bit incidentally means that there is a total of 24 bits of information in the three color channels[xxiii].
The measured values are at the lowest measured (rather than nominal) ISO setting. The camera with the lowest measured ISO setting has a slight advantage that disappears at higher ISO settings like 200 ISO. See Measurements | Dynamic Range tab for the graphs on www.DxOMark.com .[iii]
I do not shoot weddings. As I was nevertheless requested to shoot a close friend’s indoor wedding, I rented a Canon 5D3 with some fast lenses (e.g. 24mm/1.4 II and 85mm/1.2) – just to be on the safe side.[iv]
Actually Lightroom shows that some spots are overexposed. This could easily be fixed with the LR4 highlights slider, but I prefer to shown the unprocessed photo.[v]
(0.05/90)^(1/2.2)=3.3% on Lightroom’s histogram readout whereby 2.2 is the presumed gamma value.[vi]
For a presentation by Hugo Gaggioni, the CTO of Sony’s Broadcast and Production Systems division on this go to http://pro.sony.com/bbsc/video/videoChannelSearchResults.do?pageno=2&navId=4294963750&sort=relevance&srchTerm=sensor&pagerecs=12&view=grid and click on the "CCD & CMOS" thumbnail. Sony’s Exmor technology is discussed between 22:00 and 29:47.[vii]
The use of on-chip or per-column ADCs has multiple benefits. Firstly, the analog signals are measured closer to the source, thus reducing the chance of interference (EMI) associated with long wires and small signals. Secondly, the rate at which column ADCs measure pixels is reduced by roughly 1000x (image width in pixels divided by number of sampling channels). This means that the ADC can take e.g. 1000 times longer to measure each individual pixel. As mentioned in http://en.wikipedia.org/wiki/Analog-to-digital_converter , “There is, as expected, somewhat of a tradeoff between speed and precision.” So a radical reduction in the speed requirement can be expected to improve precision and thus noise.[viii][ix][x]
The CMOS chip is made of silicon, but the wiring is done using copper instead of aluminum. Sony already uses this technology.[xi]
0 dB doesn’t mean zero signal. It means signal and noise are equally strong.[xii]
Ln(sqrt(16 MPixels/8 MPixels))/Ln(2) = 0.5 EV. There are smarter ways to calculate this, but this works.[xiii]
Choosing a reference resolution of 16 MPixels instead of 8 MPixels would thus decrease all the DxOMark Print mode dynamic range numbers by 0.5 EV.[xiv]
The benchmark doesn’t depend on the actual steps (e.g. 1.0 stop or 1/3 stop) in which a user can adjust the ISO setting. Results for intermediate values are generated by interpolation.[xv]
Strictly speaking, the definition doesn’t allow you to express the Low-Light ISO behavior of a camera with a small enough sensor if the camera fails to meet one or more of the three criteria at its base ISO setting. But one of the tested models (Panasonic DMC FZ28) actually has a Low-Light ISO rating that falls below the (both nominal and actual) ISO range of the camera. So apparently this benchmark accepts extrapolated results.[xvi]
Pruning down the sensor would reduce the resolution. But until the pixels get very small, the DxOMark results should be roughly independent of resolution.[xvii]
Creating an array of about 30 tiled Q sensors would result in a full-frame sensor which, in theory, would outperform the reigning Nikon D3s! And - assuming one could do the tiling seamlessly and could handle all the resulting data - would result in a 360 MPixel übersensor. Or you could make a 700 MPixel medium-format sensor that would outperform all full-frame and medium-format sensors. Actually this may put Canon's 120 MPixel "proof-of-concept" APS-H sensor (August 24th 2010) into perspective: when scaled from APS-H to full-frame, the Pentax Q technology would give 200 MPixels. I don’t know what the purpose of Canon’s proof-of-concept sensors was: convince Canon management about scaling laws, test a manufacturing technology or test the waters for niche applications?[xviii]
The value we get here should be equal or greater to the value provided by DxOMark: we are only checking one of the three rules here.[xix]
Often the actual sensitivity is lower than expected (by e.g. almost 1 stop on the PEN E-PL5). This helps the camera look better in benchmarks.[xx]
See the Measurements à Color Response tab → Sensitivity metamerism index, ISO 17321. Because SMI is ignored in the DxOMark Camera Sensor score, is unrelated to noise, and is really hard to explain we won’t discuss it here.[xxi][xxii]
In particular, DxOMark's analysis is that Color Filter Array colors that have too much overlap in their transmission spectra increase chroma noise. Too little overlap decreases chroma noise at the cost of more luminance noise. This is an example how the details of a benchmark can impact design choices.[xxiii]
It doesn’t mean that each channel is sampled at 8 bit: each channel is typically sampled at 12-16 bit. The actual formulas for Color Depth reflect the amount of noise in each channel and are too complex to explain here (integrals).