The relationship between the incoming light on the sensor and the generated digital RAW values is usually well-known, and it is possible to make measurements from these images. By contrast, RAW converters are fairly complex. In particular, values at different positions are mixed together, making measurement extremely difficult.
Further, commercial RAW converters all aim to produce the best image quality possible, so they include many complex steps to cancel out known defects in the camera for which they are designed. For instance, noise reduction algorithms are completely necessary to obtain images with acceptable quality in low-light conditions. Another well-known example is edge sharpening, which is used to make images look crisper.
Such processing can be beneficial as far as image quality is concerned, of course, but obviously it can also potentially hide certain physical characteristics or flaws in the sensor or the lens. But most of all, it makes all measurements unreliable and unrepeatable, since they end up being highly dependent on the RAW converter, or on the way it is tuned.
The above illustration shows the detail of a single RAW image, processed with a minimal RAW converter (left) and a RAW converter with default setting (right). The left-hand image is actually more representative of the quality of the optics. The right hand image seems much sharper with more details, but that is due to the RAW converter. DxOMark’s intention is to describe the intrinsic quality of the lens and the sensor, and therefore measures a signal that is more like the left-hand image.
We do not claim that RAW converters do not impact image quality — on the contrary, there can be a tremendous difference in quality between RAW converters. But this means that the quality of the final image may not be representative of the intrinsic quality of the hardware part of the camera. If every user used only one single camera-embedded RAW converter with the same settings, it would make sense to measure image quality based on the final image. But this is not the case at all.
Since users can choose their RAW converter and tune the settings to a very fine degree, and since we want to evaluate the intrinsic quality of the sensor and the lens, it is only logical to perform measurements on RAW images.