On the dxomark web site, we evaluate and rank many types of digital cameras with image sensors that vary widely in pixel count, pixel size, and digital signal processing. To ensure that sensor performance comparisons between cameras are fair, it is very important both to test under identical shooting conditions and to take viewing conditions into account.

To compare cameras, we plot on the same graph the evolution of certain characteristics (SNR 18%, dynamic range, tonal range, color sensitivity) as a function of the ISO sensitivity value. Two essential elements can introduce bias in this comparison: ISO speed and resolution. For example, ISO 100 for a given camera may not be completely equivalent to ISO 100 for another camera; further, the resolution of the cameras may be different. More pixels mean more details, but also more noise in general. (See More pixels offsets noise!.) We review here the influence of ISO speed and resolution on measurement and discuss how we have dealt with them to ensure that the DxOMark Sensor scale and its metrics can be used to rank cameras regardless of their intrinsic resolution and sensor sensitivity.

• ISO speed defines light sensitivity by relating the amplitude of the output signal to the light captured by the camera.

  Let’s assume a photographer takes two different cameras to shoot a scene. Both cameras have same optics and are set to same exposure time and ISO speed values. Surprisingly, in most cases the output images will differ in terms of exposure. This means that either the exposure time or the ISO sensitivity is actually different for the presumed “same” setting.

  The ISO setting scale on regular cameras normally utilizes a small number of values (100, 200, 400, 800, 1600, and sometimes 640, 1250). ISO speed (or sensitivity) definitions and measurement protocols are precisely described by the International Standards Organization (ISO)1, as explained in the dxomark.com Measurement definition section. To make things easier for the user, the ISO standard allows for rounding up the actual sensitivity values to such numbers as 100, 200, etc.; however, rounding means that the nominal sensitivity can differ from the real sensitivity by up to 20%. Moreover, manufacturers can also use image processing to artificially change ISO sensitivity (by applying gains in the processing).

  To eliminate bias and rounding errors, DxO Labs accurately measures ISO sensitivity and uses it as the basis for plotting all other characteristics (SNR 18%, dynamic range, tonal range and color sensitivity).

  The Image Quality Database page provides links to ISO speed graphs that show the differences between DxO Labs’ ISO-based speed data and manufacturers’ values.

• Resolution varies from camera to camera, but ultimately images will be compared on similarly-sized screens or prints. On one hand, a higher-resolution camera will perform better because more pixels equate to more information for a given output surface. On the other hand, the same camera has a smaller pixel size and this decreases the SNR. Normalizing by pixel resolution makes it possible to compare actual sensor performance, similar to printing two images on the same printer.

  Consider a sensor with megapixels. Starting from this sensor, it is possible to simulate the noise standard deviation of a sensor with a reference resolution . In the formulas below, the values without subscripts are those that can be measured on the sensor (from actual shots of targets); values with the subscript “norm” are normalized values -- that is, corresponding to a sensor with resolution _.

  We give the relationship between the measured and normalized values for standard deviation, signal to noise ratio (SNR), dynamic range (DR), tonal range (TR) and color sensitivity (CS) as follows:

  

  

  

  

  

  As can be seen, high-resolution sensors will gain more SNR, DR, TR and CS when reduced to a lower reference resolution. For DxOMark Sensor, we chose a reference resolution equal to 8 Megapixels, which is a bit less than a 12" x 8" print with a 300dpi printer. However, any other resolution can be chosen, as doing so only shifts the normalized values by a constant (because the reference resolution appears only in a logarithm in the formulas above).
  What should be remembered is that doubling the resolution adds:

  • 3dB to the normalized SNR
  • 0.5 bit to the normalized DR
  • 0.5 bit to the normalized TR
  • 1.5 bit to the normalized CS.

  Also note that in DxO Labs’ Image Quality Database, one can look either at normalized or absolute results, yet in the context of the DxOMark Sensor scale, only normalized results are relevant. (Read more in our Insights section.)

Notes

1- Publication no. ISO-12232: Photography: Electronic still-picture cameras: Determination of ISO speed. The International Organization for Standardization, Geneva, Switzerland, 2006.