Expanding the DXOMARK Camera protocol: The Wide score

Wide-angle lenses are an extremely versatile tool—they are great for capturing narrow indoor spaces, and also for allowing you to squeeze that little bit of extra scene into the frame when photographing landscapes or cityscapes. For people shots, being able to switch to a wide-angle can be a lifesaver when a group is too large to fit into the field of view of a standard lens. You can also use the wide-angle effect creatively—for example, by creating a sensation of closer proximity to the subject than would be possible with a longer lens.

Any lens wider than a 50mm standard lens on a 35mm full-frame format is traditionally considered wide-angle, but 28mm and 24mm are arguably the most popular wide-angle focal lengths on full-frame cameras. Ultra-wide variants offer focal lengths as wide as 10mm.

On smartphone cameras, things look slightly different, however. Here, wide-angle lenses have been the standard for a long time, with most devices’ main cameras offering focal lengths of somewhere between 24 and 30mm equivalent; but with multi-camera setups becoming more commonplace, we have also seen the rise of ultra-wide-angle cameras on smartphones.

Huawei P30 Pro, main camera image
Huawei P30 Pro, wide-angle camera image

With its G5 model in 2016, LG was the first brand to push this feature, but other manufacturers soon followed suit, and today hardly any high-end smartphone comes without an ultra-wide lens to greatly expand the creative possibilities for mobile photographers. This is why we are now adding a new ultra-wide element to our DXOMARK Camera test protocol. Analogous to our current testing of the main camera and the tele-zoom modules, we look at ultra-wide angle performance across a range of focal lengths, light conditions, and use cases.

In addition to the wide-angle testing, we are also adding a new Night score for low-light testing and renaming the DXOMARK Mobile score to DXOMARK Camera.

(For more details about how we test and score smartphone cameras, please read our articles about the DXOMARK Camera test protocol and the DXOMARK Camera scores.)

Ultra-wide-angle test conditions

Our technicians test ultra-wide-angle cameras in bright light outdoors and under typical indoor lighting conditions, using real-life scenes with light levels ranging from approximately 1000 to 50 lux, and at 2mm focal length steps from 20mm equivalent out to the camera’s widest setting. For each test, our team captures more than 100 indoor and outdoor images across a large variety of test scenes designed to cover a wide range of typical ultra-wide-angle uses cases, and which allow us to assess a range of image quality attributes for each single image.

(Please note that the images below are illustrations and do not show the actual test scenes.)

Landscapes
Cityscapes
Group portraits
Architecture
Interiors
Wide-angle close-ups

Ultra-wide-angle image quality challenges

When analyzing the image results from our test scenes, we look at all the same image quality attributes as for the main camera, including exposure and dynamic range, color rendering, texture and detail, image noise, and autofocus performance and accuracy.

Ultra-wide-angle image quality challenges

For example, capturing a wider field of view increases the likelihood of capturing a scene with a wider range of contrast. In wide-angle images, this can often result in a slightly too-pronounced HDR effect, giving images an unnatural appearance. For a good result, a wide-angle camera has to find a good balance between capturing highlight and shadow detail, and rendering an image with natural contrast and color.

Compared to the primary camera in multi-camera setups, the wide-angle cameras in current devices also often capture lower levels of detail and higher levels of noise. However, both attributes greatly impact many types of wide-angle images—for example, landscapes and portraits, which is why they form an important part of our image quality assessment.

Detail and texture
Detail and texture, crop
Image noise
Image noise, crop

While these “classic” image quality attributes are very important in our assessment of image quality and scoring, in our wide-angle testing we emphasize image artifacts more than usual, because ultra-wide-angle lenses, due to their construction and wider field of view, can be susceptible to a variety of artifacts than lenses with a longer focal length. The number and visibility of image artifacts generally increases with the wideness of the field of view, and offering good image quality across all wide-zoom settings is a challenge for lens designers and device manufacturers.

This said, any image quality attribute can have a significant impact on the wide-angle score if it noticeably improves or degrades the image content. For example, an image that is strongly underexposed will receive a low score, even if all other image quality attributes are good.

Distortion

Wide-angle lenses generally show more distortion than standard or tele-lenses, and straight lines in a scene can often appear bent or curved in an image captured with an ultra-wide lens, especially for elements close to the edges of the frame. The effect tends to be most noticeable in architectural images and cityscapes where typical scenes include many straight lines and edges.

In the comparison below, you can see that the level of distortion can vary a lot between devices. The Huawei P30 Pro image shows the straightest lines and least amount of distortion. The Samsung Galaxy S10+ offers a wider field of view, but still controls distortion quite well. The effect is noticeably the strongest for the Sony Xperia 1  with most straight lines on the building clearly curved.

Huawei P30 Pro, least amount of distortion
Samsung Galaxy S10+, moderate distortion
Sony Xperia 1, strongest distortion

Some devices correct distortion in image processing and some even apply corrections based on the content of the frame. In the two samples below, you can see that the Xiaomi Mi 9 applies less distortion correction when a human subject is detected close to the edge of the frame, so as to avoid producing correction artifacts and to render the human subject as naturally as possible.

Distortion with subject at the center of the frame
With a subject close to the edge of the frame, the Xiaomi Mi 9 applies less distortion correction to avoid body deformation.

Face deformation

Faces that are close to the edge of the frame are also affected by distortion, resulting in a deformation of the head and facial features. This effect tends to be stronger the wider the focal length of the lens. We captured all the samples below at the devices’ respective widest focal length. The effect is most pronounced on the Galaxy S10+, which also offers the widest field of view. It’s also quite noticeable in the Huawei image but more controlled in the Sony image.

Huawei P30 Pro, noticeable deformation
Samsung Galaxy S10+, strongest deformation
Sony Xperia 1, best-controlled deformation

Loss of acutance

Corner softness or a loss of acutance in the field is often more noticeable for super-wide-angle lenses than it is for standard or tele-lenses. In the samples below, you can see that the camera captures high levels of detail and renders fine low-contrast textures nicely at the center of the image. However, the image is blurry and levels of detail very low toward the edges of the frame.

Huawei P30 Pro, texture at the edge of the frame
Huawei P30 Pro, crop
Huawei P30 Pro, texture at the center of the frame
Huawei P30 Pro, crop

Perspective tilt

When the camera is tilted upward, distortion causes vertical lines to fall in toward the center. This is perspective tilt, sometimes also called converging parallels. As with some of the previous artifacts, this effect is generally more pronounced on super-wide-angles than on longer lenses. In theory, the effect could be corrected by using gyroscopic data to determine the tilt angle of the device, but to our knowledge, no manufacturers to date have implemented such a system.

In the comparison below, you can see that all three images show perspective tilt, but the the effect is stronger on the Samsung and the Sony, which offer a wider field of view than the Huawei. The Sony image also shows very strong distortion.

Huawei P30 Pro, moderate perspective tilt
Samsung Galaxy S10+, strong perspective tilt
Sony Xperia 1, strong perspective tilt

Flare and color fringing

Ultra-wide-angle photographs also frequently suffer from common artifacts such as color fringing (chromatic aberration) and flare. Lateral chromatic aberrations occur when red, blue, and green light coming from the same point in the scene does not hit the sensor at exactly the same place. We can see longitudinal chromatic aberrations when the red, blue, and green color channels are not equally sharp.

Lens flare is generated when light, usually from bright light sources, is scattered within the lens system. Again, super-wide-angles can be more prone to this kind of artifact than other types of lenses; luckily, the effect can be pretty easily corrected in post-processing.

In the comparison below, you can see longitudinal chromatic aberrations along high-contrast edges in all three images, but they are most intrusive in the Huawei P30 Pro image. The other two devices, especially the Samsung, control this ultra-wide-angle camera artifact noticeably better.

Huawei P30 Pro, strongest purple fringing
Huawei P30 Pro, crop
Samsung Galaxy S10+, well-controlled purple fringing
Samsung Galaxy S10+, crop
Sony Xperia 1, average purple fringing
Sony Xperia 1, crop

Scores

To compute a Wide-angle scor,e we analyze all the image quality attributes listed above, including artifacts, at 2mm focal-range steps from a 20mm equivalent focal length to the camera’s widest setting and at the default wide-angle focal length. Results in the range of 16-20mm, which is most common on current devices, are weighted more heavily than those at even wider settings.

This said, a wider field of view can help increase the Wide score. For example, if two devices achieve the same score for the 16-20mm range, but one device also offers a wider 12mm field of view, this device will achieve a better overall Wide score. On the other hand, if a device delivers bad image quality from 12 to 20mm, it will score lower than a device that offers good image quality but a narrower maximum field of view.

The Wide-angle score then feeds into the DXOMARK Camera score, just like the Zoom or Bokeh score. The Wide score works like a bonus for those devices that come with the feature, but a bad result won’t reduce the overall score. Devices without an ultra-wide-angle lens will simply score zero points for Wide-angle.

The new Wide angle test can only add to the Photo and DXOMARK Camera Overall scores but cannot reduce them.

DXOMARK encourages its readers to share comments on the articles. To read or post comments, Disqus cookies are required. Change your Cookies Preferences and read more about our Comment Policy.