Chasseur d'Images, the first European magazine dedicated to photography, has chosen to rely on DxOMark Mobile technology for its new section devoted to smartphone image quality measurement. In its December issue, the magazine goes behind the scenes at DxO Labs to explain how DxOMark tests camera phone image quality by following a procedure as rigorous as that applied to conventional digital cameras and lenses.
The time has finally come to start testing them just like real cameras!
A telephone that shoots photos using a 41 Mpix sensor, another that produces hand-held panoramas, camera phones that film special moments in Full HD and make sharing photos and videos as easy as sending a text message... No doubt about it, the camera phone has come of age! In its labs, DxOMark tests the image quality of mobile phones according to a testing protocol as rigorous as that used for cameras and lenses, and confirms that certain mobile phones have already surpassed some of the compact cameras we were admiring only months earlier. Without denying the merits of using real cameras, the time has come to take a closer look at these devices… with a photographer’s eye!
Just a few steps from the TF1 tower, in the muted atmosphere of a room painted entirely black, a technician goes about his business amidst stands, lightboxes, color charts, numerous measuring tools, and funny pictures composed of both matte and shiny colored objects. Tomorrow he will cross Paris, following a strict itinerary from Pompidou Centre to the Eiffel Tower before heading back to the lab to take a series of portraits, also very carefully controlled. We are with the publisher of DxOMark, and we have just arrived smack-dab in the middle of testing the camera of a new mobile phone that will soon appear in store windows, and whose initial assessments are already very promising.
DxOMark’s decision to test camera phones is based on a simple observation: more than a billion mobile phones have been sold throughout the world, of which 640 million are smartphones. Just in France alone, 4.5 times more camera phones have sold than digital compacts and 27 times more than DSLRs! Camera phone image quality has become one of the principal choice criteria for buyers. Until now, however, it has never been rigorously evaluated. With its enormous experience of 2,500 camera-lens combinations already tested and more than 200,000 available measurements, French company DxO Labs has launched itself into this vast undertaking of objectively measuring and making comparisons of the image quality between devices that, up until now, were never expected to be players in the field of digital photography.
The challenge of testing camera phones was not easy to overcome, however, because contrary to lenses and DSLRs, whose components can be measured separately, the camera phone is an integral unit — a homogenous but closed ensemble which for the moment does not allow users to control the shooting parameters. No exposure control, no shutter control, nearly always used in autofocus mode, and all images saved as JPEGs affected by the same specific flaws (including “color shading”), camera phones deliver results that are very dependent on the conditions of use, which makes objectively comparing different models extremely difficult to achieve.
This observation forms the basis for DxOMark’s design of a rigorous procedure that consists of testing these devices under perfectly-controlled conditions and of combining these laboratory measurements with a perceptual analysis of natural scenes.
Color, exposure, contrast, digital noise and detail reproduction, artifacts, autofocus, flash, and video... everything is measured, compared, and quantified for the sole purpose of reaching an assessment about image quality that can fit on a single page. Such a single-page summary is based on 30-odd pages of data obtained from shooting 400 photos and 20 videos shot under lighting conditions and reproducing scenes typical of those that the device will face once in the user’s hands.
Designed to allow for fast and easy shooting, camera phones have virtually no setting controls and adapt automatically to the subject being shot. This strategy is fine for the general public, but places the technician in a delicate situation when it comes to evaluating each model’s performance, because the slightest variation in framing or lighting can lead to a different result.
|They test camera phones in the DxO lab! Shots of patterns and physical measurements of color rendering and lens performance are complemented by the perceptual analysis of typical subjects. Here, a technician shoots under controlled lighting, and the shots will be analyzed to measure the way in which a camera phone renders colors under different shooting conditions. Numerous other tests are performed that include shooting controlled pattern charts as well as real scenes in order to analyze and understand all the parameters that lead to the fact that… some photos are better than others.|
• Light: quality and quantity. DxOMark’s testing protocol addresses this problem by simulating normalized lighting conditions that are representative of those for which the camera was designed. During testing, the camera phone shoots under different lighting conditions (D65 daylight, tungsten A, TL84 fluorescent, and Clear White) and progressively stronger lighting intensities: 5 lux (streetlights), 20 lux (restaurant), 100 lux (well-lit room), and 700 lux (grey cloudy sky). Such use cases can say a lot about the performance of these inseparable ensembles of sensor and embedded software, and can reveal which camera phone, excellent in broad daylight, will see the quality of its images collapse in the face of low-light scenes.
For this test, in addition to the equipment used to modulate the light quality and quantity, DxOMark photographs a standardized scene composed of a multitude of objects. Analyzing these images allows for comparing the way the camera phones reproduce colors, textures, dark zones or uniformly light-colored areas. Certain points are measured, while others a visually compared. This analysis reveals the care with which the manufacturer took to optimize the image before its transformation into a JPEG file: your photo!
• In the field.... The test is then complemented by measurements of shots taken outdoors of subjects chosen, depending on the kind of photographic challenges they contain. On one such image, the engineers are looking for artifacts in bright patches or to see how the camera phone balances the delicate compromise between digital noise smoothing and preservation of fine details. For another subject, they will analyze color fringing or evaluate the dynamic range — that is, the ability of the camera to render subjects with strong differences in contrast.
• Under “normal user” conditions. To ensure the reliability of its perceptual measurements, the camera phone undergoing testing is evaluated at the same time as a reference model under the same conditions. Any variation of the procedure is noted as well.
But the “DxO techies” aren’t just wedded to the laboratory, but also know how to shoot photos under “normal user” conditions, so the images are evaluated using the equipment for which the end-product is intended: HDTV, computer screen at 100 %, websites.... This is not an excess of caution, but instead a reliable way to highlight flaws or imperfections that would otherwise be invisible on the tiny screen of a mobile phone.
For nearly 10 years, Chasseur d’Images tests have relied on the DxO Analyzer tool to measure the quality of lenses and digital cameras. These tests are based on the same set of physical measurements that also form the basis for DxO Optics Pro, DxO’s software for processing RAW image files. The DxO Labs company provides these tools to manufacturers of sensors, lenses, cameras, and camera phones to allow them to measure product performance both during and at the end of the manufacturing process, and for certain models, even at the chip level; all this to say that DxO’s expertise is important! So it is simply logical for DxO Labs to apply this same know-how to camera phone measurement and to complete its perceptual analyses by using the same physical measurements that it applies to lenses and digital cameras.
In the DxOMark lab, we found camera phones facing the patterns we’ve come to know well: colored checkerboard, dot chart for measuring sharpness, resolution, and distorsion. Same conditions, same measurements... but with certain adaptations that take into account the specificities of camera phones (i.e., AF not refocusing between two images, or noise smoothing that changes depending on the level of light level, for example).
DxOMark by DxO LabsResolutely dedicated to tests and measurements, the DxOMark website has become a favorite reference for those who want to know more about sensors, lenses, cameras... and from now on, camera phones as well! There you can find test curves and figures about a large array of equipment, all gathered together in a database that is constantly enriched as new products arrive on the market. The only obstacle for French speakers: the content is in English only. A comparator helps users see the strong and weak points of several devices all at once. More to explore at www.dxomark.com
Before delivering their full 30-odd page report of figures, curves, and graphs, the engineers must take hundreds of images under exacting conditions. The overall score for a device is based on seven test categories: color, exposure, contrast, noise, texture, lens quality, and autofocus.
• Color. This test consists of photographing a Gretag-Macbeth chart under known lighting conditions, then comparing the images against the chart’s 24 patches so as to measure the accuracy of the tone rendering. The results are reported in the form of a CIE-Lab graph on which colored points symbolize the true colors (i.e., the colors that the camera should have faithfully rendered), and arrows show the gap between as well as the direction of the drift away from the true colors.
DxO doubles up on this measurement with a perceptual test consisting of taking a photo under tungsten lighting of a person standing against a uniform background while holding a Gretag chart. This way one can see the accuracy of the color rendering and the uniformity across the field. Because of their extreme compactness, camera phones are often subject to a particular flaw that presents itself as a gradient moving from pink in the center to green at the edges — very annoying when the bride is wearing white!
• Exposure and contrast. Lacking the settings present on DSLRs and advanced compacts, the user has to rely on the camera phone’s automatic exposure. Testing this consists of a shooting series of images of typical subjects that are then visually analyzed to assess the camera phone’s overall exposure of the scene and its ability to restore large differences in contrast. Let’s not forget that the photos for these cameras are saved as JPEGs, so the potential effectiveness of subsequent corrections is more limited than for RAW image files.
• Noise and texture: The impossible compromise. Noise to a digital photo is what grain is to a film photo: at the sensor’s limits, images become “flaky,” noisy, and are rendered as an amalgam of pixels. To limit this phenomenon and to stave off the appearance of noise, manufacturers employ smoothing algorithms… but too much smoothness can destroy the finer details.
DxOMark measurements aim to understand and evaluate the processing carried out before image recording, in other words, to detect any action of a detail-sharpening filter and/or denoising feature. The “judge” in this instance is a picture of an array of dead leaves — a subject composed of random textures such as one would find in real life (for example, hair, wrinkled clothing, foliage, etc.).
In practice, digital noise appears as the light decreases, and it is more noticeable in the dark zones of an image. But it can also appear in a sky or light-colored zone if, faced with a lack of sensitivity, the camera phone software boosts the sensitivity instead of extending the exposure time!
Top and far left, an image taken with an iPhone 4s under tungsten lighting at 20 lux.
Right and below, the same image taken with the same camera, but in daylight at 700 lux.
The analysis of each of these multiple reference objects in the test picture permits a first assessment of the image quality delivered by the camera phone under lighting conditions that vary both in quality and quantity.
Additional batteries of measurements of outdoor and studio shots, followed by physical measurements of shots of patterns and color charts in the lab, complete the DxOMark Mobile testing protocol.
We understand how difficult it is to quantify this kind of problem: the camera phone is a finished single piece that produces images according to the preferences of its designer, which can haphazardly change from one model to another or from one update to another! One manufacturer confronted by a less sensitive sensor can obtain a better performance by prolonging the exposure time (thus providing more photons to the sensor!),with all the resultant risks of introducing blur. Another might activate the flash or shorten the exposure. Yet another might choose compromises that depend on the lighting, sometimes by either boosting the details or smoothing the noise, and other times by a kind of combination of the two- an ideal solution that is complicated to develop and is penalized by computation time.
These choices don’t escape the DxO technicians, and based on usage, they count in the final grade.
• Lens quality. Camera phone lenses are tested according to a protocol similar to that for regular camera lenses. Here, we are on familiar ground, since these are the same kinds of measurements that we have produced ever since our CI lab started using DxO tools. It all starts by photographing a dot chart, followed by using Analyzer software to analyze the images obtained.
DxOMark measures the principal lens flaws: chromatic aberrations, distortion, vignetting at different points (center, edges, and corners), just as for a DSLR lens. This is all without forgetting, of course, that the lens, sensor and the software are inseparable, and so DxO may end up analyzing images that have already been corrected to some degree prior to being saved as a JPEG, if the manufacturer has included such technology.
• When there is little light.... After a beginning of being deprived of flash, camera phones have since been equipped to produce photos in complete darkness and to recover poorly-lit subjects. While they don’t provide the scope or coverage or quality of DSRL flashes, DxO faithfully tests them anyway, not only by measuring the intensity of the flash, the light distribution, the white balance, the repeatability, but also the color fidelity, gradients, and exposure of the images when shot with the flash alone or in combination with other lighting. In our opinion, the score obtained for this criterion is important when choosing a mobile phone which, as a companion to a “mobile photographer,” may be called upon to operate under the the most difficult lighting conditions.
• Autofocus. This feature of camera phones varies enormously from one model to another. At first, certain audacious descriptions deliberately confused autofocus with the absence of focusing, since the lenses were set on the hyper-focal distance. The arrival of more powerful sensors brought about a more precise adjustment and made Autofocus more widespread — but with important differences. For some models, the AF does not work at all light levels; for others, the user can indicate a reference range;face detection software facilitates exposure adjustment for some others. These many variants require a detailed examination of the device so as to understand how it functions, and forces testers to be extremely vigilant.
Tests follow the norms described in ISO 12233 and are based on shots of a subject taken at progressive frequencies and different lighting situations, so as to measure the detection capacity of the autofocus, along with its accuracy and its repeatability for 30 successive shots (with defocusing between each image) in auto mode and in forced (tap-to-focus) mode (if available).
|Here, one of the most sensitive tests for camera phones: these random textures allow for analyzing the restoration of fine details, digital noise, and a number of other issues that impact sensor performance, image processing, and even the way images are compressed before being saved as the JPEG file that will become... your photo!|
Installed on a computer-controlled hexapod, the camera phone is then subjected to the last battery of measurements. DxO tests the stabilization by subjecting it to cycles of vibrations that vary in intensity and duration. The measurement is automatic and is based on DxO Analyzer v5 processing algorithms. DxOMark’s protocols have revealed certain hitherto unsuspected deficiencies, including some manufacturers’ deliberate disabling of certain automations in low light, when the processor can no longer process all the data streams.
Color, exposure, texture, noise, autofocus, and stabilization are physical measurements; artifacts, however, are evaluated perceptually.
At the first presentation of DxOMark Mobile test results, DxO CEO Jérôme Ménière created a buzz when he displayed a chart comparing the overall scores obtained by three famous cameras and one camera phone: 77 for the Canon PowerShot G9, 88 for the PowerShot S100, 95 for the reflex Nikon D700... and 81 for the Nokia 808 PureView! These scores do not negate the different possibilities offered by each of these devices, but rather present a new reality: in terms of image quality, the best camera phones can now compete with conventional cameras.
The market is not mistaken: all the studies show that the quality of photos and videos has become one of the most important choice criteria for buyers of multimedia cell phones; on the flipside, sales of digital compact cameras have fallen by more than 40%. We also note that that many people are never separated from their cell phones, making it likely that they will rush to use it when the opportunity to shoot a picture or video arises. Always ready to snap and share images, the camera phone is an increasingly bigger part of the photographic landscape. For Chasseur d’Images, the time has come to consider camera phones on par with digital compacts, and to faithfully present the advantages and disadvantages of these two families of tools. — Not to proselytize either one, but to consider their respective image quality, possibilities and limits. As such, DxOMark Mobile’s tests represent a significant contribution, as they allow our journalists to base their conclusions not just on reliable, comparable, and repeatable measurements, but also on an objective assessment of the quality of the results.
|This graph shows the comparison of the image quality among two digital compacts, a DSLR, and the Nokia 808 camera phone (41 Mpix in PureView mode).|
For the first time we have at our disposal the data that will let us know and understand exactly how these devices function in the face of all the different kinds of subjects they will be called upon to shoot. We can therefore separate our product reviews into two important parts: on the one hand, an analysis of the product, its features and ergonomics, and what it brings to the field; on the other hand, the quantification of its image quality results.
Up until now, we have been used to the brief descriptions of camera phones that quickly highlight the number of pixels, aesthetic features, and gadgets; from now on, we will know everything that the specs don’t tell us….