DxOMark Camera Sensor

Friday, January 11, 2013

Camera Article
Intro | DxOMark Score | Portrait, Landscape and Sport | How fair is DxOMark ?

Four top-level graphs

After this somewhat abstract and lengthy introduction, let’s bring on more real benchmark data for actual cameras.

Figure 2.  DxOMark Camera Sensor score plotted against sensor size, price, launch date and MPixels.

Figure 2 shows the DxOMark Camera Sensor score along all four vertical axes. The DxOMark Camera Sensor scores are currently between 27 (for an old Panasonic model with a tiny sensor) and 96 (for the Nikon D800E with its full-frame sensor). Future scores could and will likely exceed 100 at some point. The DxOMark Camera Sensor score is based on three more detailed measurements which we will discuss later (“dynamic range”, “ISO” and “color depth”). This is what I called the 2nd level of the 4-level data pyramid. But for now, we are still at the 1st level: a single score.

Don't get hung up on score differences of only a few points: 5 points is roughly the smallest visible difference in actual photos (DxO says it is equivalent to 1/3 stop). The measurements themselves appear to be repeatable in DxO’s lab to within one or two points[i].

The four graphs shown in Figure 2 respectively show:

a. the impact of the sensor's physical size on the top-level score,
 b. the correlation between the overall score and the (rough) price of the camera,
 c. how digital cameras have evolved over the past 10 years, and
 d. how image quality is related to sensor resolution (= MPixels).

To save you some scrolling and squinting, each of these four graphs in Figure 2 will be shown enlarged when it is discussed.

Sensor size impact on image noise

Figure 2a. DxOMark Camera Sensor scores for different sensor sizes.
This is one of the graphs shown earlier as part of Figure 2.

The horizontal axis in Figure 2a represents sensor size relative to a "full-frame" sensor (24´36 mm). A relative size of 0.5 thus means that the sensor’s diagonal is half that of a full-frame sensor and that the crop factor is 2.0× compared to a full-frame sensor. Sensors with a relative sensor size of 0.5 exist and happen to be called “Four Thirds”[ii].

Figure 2a shows (from left to right):

  • so-called 1/2.33” sensors used in the Fujifilm X-10 and Pentax Q (crop factor of 5.6´),
  • so-called 1/1.7" sensors (5.7´7.6 mm) also used in the Canon S110 (crop factor of 4.5´),
  • so-called CX sensors with a crop factor of 2.7´ as used in Nikon 1 and the Sony RX100,
  • so-called Four-Thirds and Micro Four-Thirds cameras, both with a crop factor of 2.0´,
  • APS-C size sensors with a crop factor of 1.5× or (Canon only) 1.6´,
  • Canon’s former APS-H size sensors with their 1.3´ crop factor,
  • full-frame cameras (24´36 mm, with a crop factor of 1.0´), and
  • medium-format cameras (crop factor of 0.8´ to 0.64´).

The smaller dots represent older models and the color scale[iii] represents sensor size: orange are the relatively tiny sensors, 4/3 and APS-C are shown in shades of green, cyan is mainly Canon's 1.3x EOS 1D series, blue is for full-frame, and purple, magenta and red are "medium-format" sizes.

Figure 2a shows some interesting information:

  • The best scoring cameras are the Nikon D800 and D800E twins (highlighted in white). Both are full-frame cameras (blue dots). The D800E scores 1 point higher, although that difference is negligible.
  • The 1.5´ APS-C market segment is very full. Especially if you include the neighboring 1.6´ APS-C models by Canon.
  • Some newer 1.5´ APS-C products score between 78 and 82 points (just like Canon’s best full-frame models). High-scoring APS-C models currently use sensors supplied by Sony:
  • For APS-C sized sensors, there is no significant performance difference between SLRs, Sony’s technology with a semi-transparent fixed mirror[iv] or premium mirrorless models such as the Sony NEX-7[v].
  • Except for the 1/1.7" segment, none of the Canon models are currently best-in-class[vi]. This includes the recent Canon Powershot G1X and G15, and the EOS 650D, 5D Mark III and 1Dx. Canon is lagging behind particularly in its Dynamic Range at low ISO. Canon’s competitive 1/1.7” models incidentally use Sony sensors.
  • Some sensor sizes are performing well given their size while others are lagging behind:
    • Recent 1/2.33” and 1/1.7” sensors are scoring well compared to their size.
  •  
    • The Sony RX100 (CX or 1/1” format) is scoring well compared to its size.
    • The (Micro) Four Thirds format has been dormant until recently. The two new well performing models are the Olympus PEN EPL5 (72 points) and the Olympus OM-D E-M5 (71 points).
    • The 1.6´ format is struggling as well, but is a Canon-only format. So it reflects the fact that Canon is currently not state-of-the-art in terms of sensor noise.
    • The 1.5x APS-C format is doing well in some models due to the use of Sony sensors and likely due to the extreme competition.
    • The full-frame format is doing very well. The five highest scoring full-frame cameras are incidentally all by Nikon.
    • Medium-format is underperforming in most (but not all) respects.

Price versus image quality

Figure 2b. DxOMark Camera Sensor scores for cameras across the price range.

In Figure 2b we can see:

  • DxOMark’s database covers roughly a 100:1 price ratio ($400 - $40k).
  • On a linear scale, 85% of the cameras would be squeezed into the leftmost 10% of the graph. On a logarithmic scale these 85% end up in the left half of the graph.
  • Some models are old (small dots) and are no longer manufactured. Actual prices on the used market will vary from the nominal prices shown.
  • Increasing your budget buy you more image quality up to around $3000. Above $3000, you may actually end up paying for high speed (e.g. Nikon D4), extreme resolution (e.g. a Phase One IQ180 digital back) or a legendary brand name (Leica).
  • The Nikon D800/D800E twins (3 k$, 36 MPixels) challenge 40 MPixel medium format cameras costing 10-30 k$. The D800 twins match these medium format models w.r.t. resolution, and win w.r.t. flexibility, portability and price. According to the benchmark results, the D800(E) also wins in other image quality parameters like noise and dynamic range[vii].

Older versus newer cameras

Figure 2b. DxOMark Camera Sensor scores for cameras across the price range.

The data shown in Fig. 2c is essentially the same information shown in Figure 1, but with less annotation.

Having too many MPixels doesn't really help

It is important to realize that

 

Instead, the score is a measure for achievable print quality for typical use cases where print quality is not limited by sensor resolution. Obviously this assumes sufficient lens quality, and that the photographer knows how to get the most out of the equipment.

So… why did DxO decide not to factor sensor resolution into the DxOMark Camera Sensor score? Firstly, this is because current sensor resolution is generally high enough for producing gallery-quality prints. Secondly, lens sharpness (rather than sensor resolution) is often the weakest link when it comes to achievable resolution. 60 line pairs per millimeter is considered an exceptionally good lens resolution. D-SLR sensors have a typical pixel pitch of 4-6 µm, corresponding to 125-90 line pairs per millimeter.

As this is important, let's double check this by estimating what resolution is needed for high quality prints. At 250 DPI print resolution, A4 (8.3"×11.7") or A3 sized prints respectively require 5 and 10 MPixels. In these estimates we assumed some white space around the actual photo. Because 250 DPI equals about 100 pixels per square millimeter, our eyes will have a tough time assessing this sharpness without a loupe. In my own experience, a  6 MPixel Canon 10D can produce great A3 prints[viii] without any fancy sharpening acrobatics - providing that you used fine lenses.

These numbers are a bit surprising when you consider that sensors only measure one color per “pixel” and thus lack information compared to true pixel as defined on screens or prints (due to the Bayer mosaic[ix]). Fortunately the camera industry is quite good at reconstructing the missing color information using fancy demosaicing[x] algorithms. It also helps that our eyes are not especially good at seeing abrupt color changes unless these coincide with sudden brightness changes. If you want to check this, the comma in the middle of this sentence is actually blue instead of black. You will probably find this hard to see against a white background. So even when viewed at "100%", images taken with good lenses can look surprisingly sharp.

But wouldn't we need more pixels for say A2-sized prints (15” × 20” print area)? Not necessarily: if you view bigger prints from a larger distance in order to see the image in its entirety, the required resolution doesn’t increase further and stays at the level of the (angular) resolving power of our eyes.

You will be hard-pressed to buy a new SLR camera below 16 MPixels (see Figure 3), so those extra MPixels enable you to crop your images during post-processing if needed - again assuming your lenses are top-notch. And high resolution numbers also helps impress your (male) friends at the bar or possibly the customers at your gallery.

Figure 3. Launch dates versus MPixels. The lines represent the various Canon product lines.

Figure 3 shows how resolution increased over the years. As indicated by the title at the top, this particular diagram shows than the 187 cameras tested by DxOMark as well as various still untested[xi] cameras.

The MPixels shown along the vertical axis of Figure 3 corresponds to the general public's measure for image quality. The rather inaccurate view that “more MPixels means higher quality” can be easily disproven by comparing Figure 2 (image quality) to Figure 3 (MPixels). For example, take the orange Canon Powershot G-series: going from the G10 to the G11, the resolution was reduced from 14.7 to 10 MPixels while image quality improved if we assume that A3-sized gallery-quality prints are more than enough for the target users.

Other highlights of Figure 3:

  • The MPixel record is currently a tie between three medium format 80 MPixel digital backs. From an optics and sensor perspective there is no reason why the resolution of medium format cameras cannot be increased. The Nikon D800(E) and even the unruly Nokia 808 PureView camera phone provide the same 40 MPixel resolution as the bottom of the medium format market.
  • Despite some speculation that the megapixel race might be over, and Canon’s relative restraint in recent models, current camera models typically provide 16-24 MPixel resolution, notably including the $700, 24 MPixel “entry-level” Nikon D3200.
  • Canon replaced its 1D Mark IV (16 MPixel; 10 fps) and its 1Ds Mark III (21 MPixel; 5 fps) with a single model: the 1Dx (18 MPixel, 14 fps). The drop in resolution created an initial stir among 1Ds Mark III owners. The stir could have been avoided if the 1Dx had provided 21 instead of 18 MPixels, but at 14 fps the resulting data rate was apparently too hard to handle.
  • Nokia’s PureView 808 is a 41 MPixel camera phone with a modest CX-like sensor size. It obviously doesn’t try to compete with a 36 MPixel Nikon D800 or a 40 MPixel Hasselblad, but uses the excess resolution of the relatively large Toshiba sensor to implement high quality digital zoom. When zooming is not required, the pixels are averaged to output 5 or 8 MPixel images with acceptable noise levels.

But having too many MPixels doesn’t harm either

More MPixels imply larger image files, thus slowing down image processing and file transfer. But the good news is that more MPixels do increase image noise - despite a widespread belief to the contrary.

The reason for this is that when you scale down[xii] to a resolution required for displaying or printing, the resulting noise and dynamic range of the output pixels improve (assuming well behaved rescaling software). The resulting noise and dynamic range after scaling are then in theory identical to what you would have had if you had started off with a sensor with exactly the required resolution to begin with. And you may end up with a slightly sharper image as a bonus – but this is off-topic here.

Let's look at this reasoning more closely. We will essentially discuss a bit of basic algebra – but I will leave out the actual formulas because they scare away most readers.

Figure 4. Impact of pixel size on noise level.

Figure 4 shows an accurate analogy to collecting photons or the basic “particles” of light: measuring the rate of rainfall by collecting rain in cups. We might decide to measure the rainfall with a single large bowl. Or, alternatively, we could use for example 4, 16 or 64 smaller cups. In all these cases the effective area used for catching drops is assumed to stay the same[xiii].

In the example with 64 cups, I exposed these cups to a simulated rainfall that caused each cup to get 5 drops of rain during the exposure. For visual clarity I used really big drops (hailstones?) or – if you prefer – minute cups. However, for the signal-to-noise ratio the size of the cups doesn’t matter. Due to the statistics (Poisson distribution[xiv] with "λ=5", in the jargon of statisticians), on average only 17% of the cups will contain exactly 5 drops of rain after the exposure. Some cups will instead have 4 drops (17% chance) or 6 drops (15% chance), but some may contain 9 drops (4% chance) or even remain dry (0.7% chance) during the exposure to the rain.

In the following, instead of saying “λ=5” to mean that we are exposing the grid long enough that we expect – on average – exactly 5 units to end up in each cup/pixel, we will say N=5 to avoid any confusion[xv] with the wavelength of light (commonly denoted with λ by scientists).

This phenomenon explains a major source of pixel noise (“photon shot noise”[xvi]). This source is unavoidable and especially noticeable with small pixels, in dark shadows or at high ISO settings. The corresponding light level is shown in Figure 4 as a projected gray-scale image below the cups: empty or near-empty cups correspond to black pixels and full or almost full cups correspond to white pixels.

Now let's look at an array of 16 (instead of 64) cups. Each cup is 4× larger and will thus, on average, catch exactly 20 drops instead of 5 drops. But, after scaling, the measurements obviously result in the same estimated rainfall[xvii]. Due to Poisson statistics, we may occasionally (9% chance) encounter exactly 20 drops in a cup, but we may also encounter 18 drop (8%), 21 drops (9%), or 25 drops (5%). The odds of observing 4 or 36 drops are very small but non-zero. So, although larger cups will have slightly more absolute variation when expressed in drops than smaller cups, the relative variation expressed in volume of water per surface will actually decrease as the cup size increases[xviii].

The point here is that proper scaling allows us to get exactly the same signal and noise levels using many small cups (pixels) or using an equivalent surface area covered with a few large cups (pixels)[xix]. Thus a set of 4 cups will give you exactly the same information as a single bigger cup with a 4x larger surface area would have: just carefully pour the content of the 4 small cups into one big cup before measuring. Or weight all 4 cups together and subtract their empty weight.

Per-pixel sensor noise

Our cups-and-drops analogy gives a basic[xx] model of pixel behavior.

Now let’s calculate what happens when we have plenty of light falling on a light sensor. Real pixels in a full frame 36 MPixel Nikon D800(E) with 5 mm photodiodes can hold roughly 40,000 free electrons[xxi] that are temporarily knocked loose within the CMOS or CCD sensor by photons. If you apply even more light, the pixel can no longer measure the difference: this is known as “saturation” to sensor experts or “burnt highlights” to photographers.

For a high-end compact camera with 2 mm photodiodes, N drops to maybe 6,400 because of the smaller pixels. For a decent medium-format sensor, N might reach 100,000 electrons.

A value of N=40,000 for the Nikon D800(E) implies noise level fluctuations in the order of 200 electrons. This is because, for a Poisson distribution, the “standard deviation” equals the square-root of the expected average. A ratio of 40,000 to 200 gives a signal-to-noise ratio of 200:1 (typically 200 electron variation on an average measured value of 40,000; "46 dB"[xxii] in engineer-speak). This is under the best possible circumstances: it is the noise level within an image highlight at the camera's “base” ISO setting (say 100 ISO) when you are “exposing to the right” (meaning: just before saturating the highlights).

So instead of N=5, N=20, N=80 and N=320 as simulated in Figure 4, actual sensors have “full well capacity values” or N values of thousands or tens of thousands of electrons per pixel. The basic math, however, stays the same and tells us that if N=40,000, the photon shot noise levels in the highlights are imperceptible to the eye[xxiii]. For an example, see the well exposed part of Photo 1 as recorded on a Nikon D800 at 100 ISO.

Photo 1. The shadow in this 100% crop is roughly 6% gray (100 ISO, Nikon D800, 85mm, f/1.8)
From the Nikon D800 review at http://www.imaging-resource.com (used with permission).

Now let’s consider parts of an image that are exposed four stops lower (-4 EV, 6% gray) than the highlights. This holds for the blurred shadow in Photo 1. Each pixel here holds a signal of about 40,000/(2×2×2×2) electrons or N=2,500. This gives a noise level of 50 electrons and a signal-to-noise ratio of 50:1 or 33 dB[xxiv]. That level of noise is normally almost invisible, but we can see it by pixel peeping at a blurred featureless area within this 100% crop. Remember that the original image is 36 MPixel image and that we are thus seeing only 1.5% of the image. The lesson here is that even at 100 ISO, dark shadows exhibit noise that can be seen (if you deliberately go look for noise) on any camera[xxv].

We will now make matters worse by simulating the wedding photographer’s shooting without with only ambient light in the ambience of a dimly lit castle. Say this requires boosting the ISO from say 100 to 3200 ISO (see Photo 3). This means that we are underexposing the sensor by 32×!

 

They already told you that, right?

So exposing our dark 6% gray at 3200 ISO, leaves us with an average signal level of a measly 78 electrons per pixel, with a noise level of 9 electrons - resulting in a highly visible signal-to-noise ratio of 9:1 or “19 dB”.

It is worth noting that, once you have a “full well capacity” of 40,000 electrons, the rest is just plain laws of physics. These laws cannot be changed by smart engineers or overoptimistic R&D managers. In other words, the preceding calculations tell you roughly[xxvi] the upper limit of what any past, present or future digital camera can do.

Regardless of whether you use CCDs, CMOS, back-side illumination or even a future invention: a sensor can (and will) perform worse than these theoretical limits at high ISO – especially when the value of N is low - due to additional noise sources within the electronics[xxvii]. Estimates of these extra noise sources made by curve fitting DxOMark data can be found at www.sensorgen.info.

BUT... per-pixel sensor noise is not very relevant

This gets us back to “smaller pixels give higher per-pixel noise levels”. This is a fact and we showed you how the math for this works. But, fortunately for us users, per-sensor-pixel noise is the wrong metric for prints[xxviii]. Printing implies scaling to a (let’s assume ) fixed resolution of (arbitrarily!) 8 MPixels. If the scaling is done well, it exactly cancels out the extra per-pixel noise which you got by starting out with more than the required 8 million MPixels.

So the following four scenarios for reducing image resolution give you the same signal levels and the same noise levels – at least according to our simplified model:

  1. You can start off with an 8 MPixel sensor (“low resolution”) with the same total light-sensitive area as the other scenarios.
  2. You can use a 32 MPixel sensor, but combine each 4 analog quantities before going digital. This is like pouring 4 small cups into a bowl before weighing ("analog binning").
  3. You can using a 32 MPixel sensor, measuring the output per pixel and then scaling the resulting image to 8 MPixel digitally ("digital binning"[xxix]).
  4. You can use a 32 MPixel sensor, capturing all the measurement data in a file, and letting a PC scale the image to 8 MPixel.

Here is another example: a 60 MPixel sensor in a Phase One P65+ camera back should[xxx] give the same print quality and the same DxOMark Camera Sensor score as:

  • a hypothetical 15 MPixel sensor with the same physical sensor size
  • an image that is downscaled within the camera to 15 MPixels
  • an image that is downscaled within a PC or Mac to 15 MPixels

By coincidence (as I later heard from a DxO expert) DxOMark has actually tested the second scenario for the P65+ digital back: in its "Sensor+"[xxxi] mode with 15 MPixel output, it gets the same DxOMark Camera Sensor score as in its 60 MPixel native mode. This is reassuring for the validity of the scaling formulas.

I incidentally believe that a similar conclusion holds when you resolution rather than it, but that case is less relevant and harder to explain in a simple way. In essence the numbers like 8 MPixels or ratios of 4:1 are just arbitrary examples. If we had explained all this using formulas instead of examples, the number 8 MPixel would have been a parameter that would have disappeared (“cancelled out”) in the final result.

Resolution and DxOMark Camera Sensor score

To summarize, the DxOMark Camera Sensor score is "normalized" to compensate for differences in sensor resolution. In other words, the DxOMark Camera Sensor benchmark doesn't "punish" high-resolution sensors for having lots of small pixels that are individually relatively noisy. And similarly, the benchmark doesn't favor using large pixels despite their lower per-pixel noise. This is some kind of ideology or marketing manager’s claim: it is just the result of calculating how the noise level of different input resolutions would result in noise after scaling to a single output resolution.

Figure 2d. Although high resolution does not directly increase the DxOMark score,
 there is an indirect correlation because big sensors tend to have more MPixels than little sensors.

Having said all that, let’s go back to the benchmark data in Figure 2d. Despite theory explaining why squeezing more MPixels into the same sensor area impact image-level noise, Figure 2d seems to suggest that higher-resolution sensors actually have better performance than lower resolution sensors. This doesn’t support the myth that “increased resolution increases noise”: it in fact shows increased resolution providing decreased noise (=higher scores).

This is firstly because high resolution sensors might be bigger than lower resolution sensors, and bigger sensors have less noise. It is also because lower resolution sensors tend to be older, and sensor performance has increased over time. There is little interest in low resolution sensors, especially now that you can get high resolution sensors without a noise- or price penalty.

 

Again, this is no evidence supporting the “high resolution gives high noise” myth. In fact, it would support a “high resolution gives low noise” myth! In this particular case the sensors are manufactured[xxxii] by different factories and are thus not entirely comparable.

 

Answer: The pixel pitch would drop down to about 2.75 µm or below. At that resolution, lenses are generally the bottleneck[xxxiii] - so you won’t see much improvement in actual system resolution. Furthermore, at some point, pixels become so small that the assumed idealized scaling (with an assumed constant fill factor and constant “quantum” efficiency) will no longer apply: four 2.5×2.5 µm pixels together would capture less light than one 5×5 µm pixel (because of wiring that gets in the way, mechanical overlay tolerances on filters, “fill factor”, etc). The resulting decrease in signal-to-noise ratio would negatively impact the DxOMark Camera Sensor score.

Bigger lenses for bigger sensors? ()

To summarize: larger sensors (rather than larger pixels) have less image noise. This is because a larger sensor area can capture more light – more or less regardless of the number of MPixels the chip’s surface has been divided into.

But in order to capture more light using a larger sensor, you need a physically larger lens to capture more light to maintain the same exposure setting[xxxiv]. Here is a quantitative example:

  1. Let’s compare a 105 mm f/2.8 lens on a full-frame camera to a fancy medium-format camera with a sensor that has twice the area of a full-frame sensor. Certain Hasselblad cameras have a crop factor of 0.70 (2:1 area ratio) that fits the bill nicely.
  2. If we manage to mount the 105 mm lens on the medium-format camera, it may not fill the larger (1.41× = √2) image circle. But assuming it did, we would capture a wider field of view – which is not a fair comparison. So we need to switch to using a 150 mm lens to compensate for the crop factor of 0.70´. This restores the original field of view. And we need to make sure the design of the 150 mm lens creates a sufficiently large image circle.
  3. If the 150 mm lens is also f/2.8, we would get the same exposure times as the 105 mm f/2.8 full-frame lens. But f/2.8 at 150 mm requires an effective diameter of the front lens that is 1.41× larger than a 105 mm f/2.8 lens. This is related to the way f-stop numbers are defined: f-stop = focal length / effective lens diameter.
  4. This means that the diameter (or area) of the front lens has increased proportionally with the size (or area) of the image sensor.

This sounds credible: bigger sensors require “bigger glass” assuming that you want to use the same shutter speeds at the same ISO setting. Alternatively, you can use a 150 mm f/4 lens instead of the 150mm f/2.8 lens. Either you underexpose your image 2×, and thus end up with no noise level improvement over the original full-frame sensor.

Finer points about sensor scaling ()

Note that, in the previous section, we decreased the depth of field when switching from 105 mm f/2.8 to 150 mm f/2.8. This is consistent with convention wisdom that “larger sensors have smaller depth of field”.

But this only applies if there is a good reason to stick to f/2.8. In February 2012, Falk Lumo argued that, for example, a 105 mm f/2.8 lens should be compared or considered “equivalent” to a 150 mm f/4 lens on a 1.4x larger sensor. This may sound odd, but keep in mind that we tend to consider the 105 mm lens equivalent to a 150 mm on the larger format as they give the same image. So it wouldn’t make sense to automatically assume that setting all numbers the same across sensor sizes automatically gives you a fair comparison.

So, in his white paper[xxxv], Falk addresses the question whether it is feasible to make images taken with two cameras with different sensor sizes look indistinguishable. He calls this “camera equivalence”. In other words, Falk’s reasoning implies that the old wisdom that “larger sensors produce images with smaller depth of field (DoF)” and the newer digital wisdom that “larger sensors have better noise performance” are both an artifact of not comparing equivalent cameras.

So, to keep the DoF in our example equivalent you would need to use a 150 mm lens at f/4. But this in turn increases the exposure time by 2x compared to using a 150 mm lens at f/2.8. So Falk’s approach implies doubling the ISO value to essentially underexpose the image (while correcting during in-camera image processing). This in turn cancels out the noise benefit of having a large sensor. This obviously won’t make the large sensor owner happy, but does meet Falk’s goal of comparing equivalent cases. Once you reached that plateau of enlightenment, you can then presumably understand the subtle engineering tradeoffs which make one camera size fundamentally more attractive than another.

Table 1. Five in theory indistinguishable camera configurations

Let’s test the implications of this theory by simply applying it across a 16´ range of sensor sizes. We arbitrarily chose a light telephoto lens and a 25 MPixel camera[xxxvi]. This gives us the above table of cameras that would produce images that should be indistinguishable – even in a forensic lab (no wisecracks about EXIF fields please – we are doing high science here kids!). The red numbers are values that are impractical or impossible to design.

One conclusion is that we could take a workable smaller reference camera and easily produce the same results on a camera with a sensor. But this “emulation” comes at a cost: we may be using a f/4 lens at f/8 just to get the same depth of field. And we would be using only 25% of the 160 k-electron full-well capacity (by operating at 400 ISO instead of the native 100 ISO) just to keep the shutter speed the same. So we are essentially either under-utilizing a fancy larger sensor camera or assuming a less technology challenging larger sensor camera.

Using a sensor has the opposite problem: it demands a lens on the small camera that is unrealistically  fast. And it assumes we have the technology to achieve unchanged full-well capacities within smaller pixels. If we did have that technology, we could presumably have made the full-frame camera perform better.

So all this confirms in a roundabout way that large sensors can outperform small sensors or can alternatively match them using more straightforward technology.

Falk Lumo’s equivalence criterion shows you with scientific rigor that you might spend more on a larger camera without achieving any visible benefits whatsoever. But if you maintain f-stop values with increasing sensor size (and accept or even welcome the smaller DoF) you can conceivably get better image quality using a larger sensor.


[i]

 The repeatability of the score can be estimated by comparing the scores for virtually identical cameras. These pairs of twins include:

  • a pre-production Canon 550D as well as the actual production model
  • the Canon S95 and G12 models (believed to have virtually identical technology in a different housing)
  • the Nikon D800 and D800E (in which part of the low pass filter has been rotated to disable anti-aliasing)
  • the Pentax K-5 II and K-5 IIs (also with a disabled anti-aliasing filter)
[ii]

 They are called Four Thirds because that is their width to height ratio. Many sensor size names in this industry are obscure at best: so called “1 inch” sensors are much smaller than one inch.

[iii]

 The scale is a continuous color gradient (Matlab-style colormap). If you want to use the same coloring convention formula to represent sensor size, contact me for help.

[iv]

 Sony calls this "translucent", but this is technically not a very appropriate term. Frosted glass is translucent. Using the right term keeps Ken Rockwell happy ;-)

[v]

 In principle, the semi-transparent mirror gives a handicap of a few points because the mirror diverts some light for the autofocus sensor. This handicap gives slightly lower scores, but is relatively minor: 70% of the light reaches the sensor. That is equivalent to loosing 0.5 stop of light. 15 points was 1 stop according to DxO, so photographing through Sony's pellicle mirror (or through a 0.5 EV gray filter) should cost about 8 DxOMark Sensor points. Adding 8 points to the Sony Alpha 55's score (73) brings the camera on par with the Nikon D7000 (80) and Pentax K-5 (82) and Sony Alpha-580 (80) which are believed to use very similar Sony 16 MPixel sensors  (likely Sony's IMX071).

[vi]

 Because Canon is the only supplier in the 1.6× APS-C and 1.3× APS-H categories, you should compare these against e.g. 1.5× APS-C.

[vii]

 Note that this may partly reflect that medium format cameras are designed for high resolution and color rendering: on a tripod, high ISO can often be avoided and in a studio there should be enough light and dynamic range can be controlled.

[viii]

 5 MPixel for A3 (with a bit of border) corresponds nicely to the 180 DPI lower limit recommended for gallery-quality prints in Luminous Landscape’s in From Camera To Print - Fine Art Printing Tutorial.

[ix]

http://en.wikipedia.org/wiki/Bayer_filter

[x]

http://en.wikipedia.org/wiki/Demosaicing

[xi]

 These include some relatively new models that haven’t been tested yet (e.g. Sony’s NEX6), some that might not be mainstream enough to ever get tested (e.g. Leica’s monochrome M8-M) as well as several noteworthy old models (e.g. Canon Powershot G7).

[xii]

 This scaling is often done automatically when you print or view the results.

[xiii]

 As sensor folks say, “they have the same fill factor" or as chip designers say "it's an optical shrink". The bowl and cup shapes shown here are horizontally scaled versions of each other, thus leading to identical fill factors.

[xiv]

http://en.wikipedia.org/wiki/Poisson_distribution

[xv]

 Prof. Eric R. Fossum (who incidentally invented the CMOS image sensor) asked permission to use this article as reading material for his students and requested the change.

[xvi]

 If you have the time and courage to dive deeper, there is a tutorial series at www.harvestimaging.com that quantifies numerous sources of sensor noise. It is by Albert Theuwissen, a leading expert on sensor modeling. I created a synopsis of this 100-page series in another posting (http://peter.vdhamer.com/2010/12/25/harvest-imaging-ptc-series/).

[xvii]

 When expressed in millimeters, or in water volume per unit of area.

[xviii]

 Cups that on average catch N drops during the exposure to rain will on average have a standard deviation of sqrt(N) drops. To estimate the rainfall ρ we get ρ = N × drop_volume / measurement_area. The expected value of ρ is independent of cup size. And the variation of ρ decreases when larger cups are used. In Figure 4, ρ would be the depth of the water in the cups if the cups had been cylindrical. So as N is increased (bigger cups or longer exposure), the Signal-to-Noise ratio improves. But ultimately we care about how hard it rains, rather than caring about droplets per measuring cup. If you measure rainfall with a ruler to see how deep the puddles are, you will get a result that doesn't depend on puddle size, and the noise due to drop statistics will decrease for larger pools of water.

[xix]

 If you still don't believe this, go read DxO's white paper "Contrary to conventional wisdom, higher resolution actually compensates for noise “. http://www.dxomark.com/index.php/Publications/DxOMark-Insights/More-pixels-offset-noise!

[xx]

 To make the model more complete, you could:

  • Measure the amount of water in the cup by weighing each cup. If you don’t subtract the weight of the empty cup, you have a significant “offset”. If you do subtract the weight of empty cups, the correction will not be perfect.
  • Assume some random errors when measuring the amount of water per cup. This “temporal” noise has a fixed standard deviation, and has most impact when the cups are nearly empty.
  • Assume that the cups are not perfectly shaped (“Fixed Pattern Noise”). Maybe rows or columns of cups came from the same batch and have correlating manufacturing deviations (“row or column Fixed Pattern Noise”).
  • Drill a hole near the top of each cup so that excess water from one cup doesn’t overflow into neighboring cups. The holes will have slight variations in their location or size: “saturation or anti-blooming non-uniformity”.
  • Place the cups in a tray of water. If the cups are slightly leaky (unglazed flower pots), you will get some water leaking in from the surroundings into the cups (“dark current or dark signal”). Not all cups will leak equally fast (“dark signal non-uniformity”). And at higher temperatures, you will see a bit faster leakage (it would be too tricky to emulate the exponential temperature dependency without some really fancy materials).
  • Break a few cups or their measurement scales (“defective pixels”).

The above covers all the noise sources in the PTC tutorial on www.harvestimaging.com.

[xxi]

 For info on the maximum value of N or "the full well capacity", see Roger Clarke's website. See http:// www.clarkvision.com/articles/digital.sensor.performance.summary/#full_well. More recent data derived by “curve fitting” the DxOMark measurements can be found at http://www.sensorgen.info/.

[xxii]

 DxOMark’s value for the signal to noise ratio (SNR) of the Nikon D800 at 100 ISO near saturation can be found at http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800 by selecting the “Measurements” tab and then “Full SNR”. At 100% gray, 100 ISO gives 43.3 dB. This is lower than the expected 46 dB due to curvature in the graph at high light intensities (likely anti-blooming or controlled overflow circuitry): extrapolation of the 100 ISO line between 0.2% gray and 10% gray results in an estimate of 46.7 dB. The latter value would correspond to N=46,777 electrons, which is in line with www.sensorgen.info ‘s fit of N=44,972 electrons.

[xxiii]

 You would get the same statistics when you measure rain using 2 liter pans. Two liters correspond to about 40,000 drops of water.

[xxiv]

 The actual DxOMark measurement value at 6% gray and 100 ISO is 34.4 dB.

[xxv]

 The only way to improve this signal-to-noise ratio is to increase the value of N. A drastic 4x decrease in the number of MPixels would boost N by 4´ and increase the signal to noise ratio by 2´ (square root of λ). We will see later that the impact of decreasing the number of MPixels is cancelled out by scaling statistics.

[xxvi]

 The only variable you can control the full-well capacity per unit area (bucket depth), but this number does not vary dramatically. Example: the Canon 5D2 and the 5D3 only differ by 5%. When you compensate for the difference in resolution, this is still only about 15% change in 3.5 years time.

[xxvii]

 See endnote 25.

[xxviii]

 Or, for that matter, for any other way to view an image in its entirety without zooming in and out all the time.

[xxix]

 Some cameras like the Canon 5D Mark II do this digitally. Canon calls these Raw modes SRaw and they have unusual MPixel ratios like 5.2 to 10.0 to 21.0.

[xxx]

 Note that although this scaling story holds for photon shot noise and dark current shot noise, other noise sources don’t necessarily scale the same way. In particular, some very high-end CCDs can use a special analog trick (“charge binning”) to sum the pixels, thus reducing the amount of times that a readout is required. This would reduce temporal noise by a further sqrt(N) where N is the number of pixels that are binned. Apart from the fact that only exotic sensors have this capability (Phase One’s Pixel+ technology), DxOMark’s data suggest that this extra improvement doesn’t play a significant role.

[xxxi]

http://www.luminous-landscape.com/reviews/cameras/sensor-plus.shtml

[xxxii]

http://image-sensors-world.blogspot.nl/2012/10/chipworks-reviews-full-frame-dslr.html

[xxxiii]

 This is due to a combination of lens quality and optical diffraction. For info on diffraction: http://www.cambridgeincolour.com/tutorials/diffraction-photography.htm

[xxxiv]

 Maintaining the same aperture in order to keep the same exposure time implicitly assumes that one wants to maintain the same (e.g. “base”) ISO setting. As a side effect, this lowers the depth of field and lowers the noise level on larger sensors. This is often considered desirable. But as pointed out by Falk Lumo (discussed in the next section) this is only one option.

[xxxv]

http://www.falklumo.com/lumolabs/articles/equivalence/ : Camera Equivalence by Falk Lumo

[xxxvi]

 For simplicity we assume the sensors all have the same aspect ratio or that differences in aspect ratio are cropped away in the resulting image.