Back to Tech articles
Home > Tech articles

Benchmarking home security cameras: DXOMARK tests image quality of video doorbells

Around the time that  DXOMARK was beginning to study the evaluation of image quality of home surveillance cameras, one of its new employees installed a video-camera system in his new apartment. The employee was new to the Paris region, and he was eager to try out a home security system, especially since he lived on the ground floor of an apartment complex. It quickly turned out that having the security system at home was a smart decision.
About a week after installation, the employee witnessed his apartment being burgled, live on his smartphone — he even had a conversation with the intruder! And thanks to the surveillance camera, it was all caught on video. The police intervened, and although the intruder was never caught, a video of him in the apartment was kept as evidence.

With incidents like that, it’s not hard to see why home security cameras are a big and growing business. According to a KBV Research market report, the global smart home security cameras market size is expected to reach $10.4 billion by 2026, rising at a market growth rate of 16.5% (compounded annual growth rate) during the forecast period. Security issues in developing countries are a key driver of the growth of the home security camera market, according to the research.

In an era of connectivity, this growth is in parallel with the growing number of smart home devices hitting the market every day. Most security cameras, for example, can work with Amazon Alexa, Google Assistant, and other virtual assistants.
The main reason for getting a home security camera system is to deter the possibility of a home invasion. Burglars are less likely to try to break into a house knowing that their actions are being recorded or being broadcast on a smartphone or tablet.

Indoor cameras help keep tabs on what is going on at home.
Home security cameras can help identify intruders.

While home security is the main use case for surveillance cameras, there are several secondary use cases that make a home-camera system practical to have. One particularly common use case is being able to interact remotely with a delivery person, or check on delivered packages that get left at the door. Another secondary use case is pet watching.  Now that many workers are returning to the office after having worked from home because of COVID, pets who had become accustomed to having their favorite humans at home might begin to suffer from separation anxiety. An indoor home camera system could help people check in on their pets during the workday, and they can even talk to them via the sound system. Another use case is to monitor who enters the home during the day, whether it’s the kids coming home from school with their friends or the cleaning lady.

Types of home security cameras

The main types of home security cameras are outdoor, indoor, and doorbell cameras.  Outdoor cameras monitor the activity in the yard or the driveway, while doorbell cameras allow you to see and even talk to the person at your door, whether it is a delivery person or an unexpected visitor,  even if you are not home. Indoor cameras record activities in the rooms they are placed in, such as the living room, kitchen, or bedroom—or anywhere a home intruder might be able to enter from the outside.

The choice of which type of camera to get depends on its intended use. But in general, there are some main common qualities of any type of home security camera. They must provide:

  • Good detail rendering at long distances, in the case of outdoor cameras, and at short distances, in the case of indoor and doorbell cameras.
  • Wide dynamic range
  • Large fields of view
  • Night vision
  • Audio capabilities, particularly for indoor and doorbell cameras

Challenges and limitations

Home security cameras are usually small so that they can be discreet and unobtrusive. The sensors are about the same size as those found in smartphones, meaning they are much smaller than what you find in  DSLR cameras. This presents some major challenges and limitations when it comes to image quality. Small apertures limit the flow of light to the sensor, which affects exposure and dynamic range. Small sensors mean less light is captured, which could affect the final image.

Another challenge, especially for outdoor cameras, is capturing a wide field of view.  Moreover, they use fisheyes lenses, which have smaller apertures,  to cover a wider field of view.  Fisheye lenses can sometimes create artifacts like distortion and color fringing.

The third challenge is detection, and image quality plays a role in this. The most important and basic function of home security cameras is that they must be able to quickly and accurately detect relevant motion in a use case such as a home invasion. The keyword here is relevant, and image quality from the camera is crucial in this case. If the image quality is poor, the security system could easily send too many false alerts, or worse, it might not send any at all because it did not detect anything.

The way a home security camera is installed also presents a challenge. Is it wired or battery-powered? A system that is battery-powered could be placed anywhere, but it could potentially limit the camera’s performance in terms of image processing so as to not drain the battery too quickly. A system that requires to be plugged in, on the other hand, might have to be placed in a less than ideal location and might struggle to capture good images. Other installation considerations include the ease of setup, the network, and power, application control as well as cloud usage and storage.

Another constraint on image quality is the network the security system runs on. The uploading of recorded images is subject to compression, which affects the final image, especially on battery-powered systems. The quality of the network will also affect the final image. If two bars of wifi are lost during an upload, compression artifacts will most likely show up in the final image.

Home surveillance cameras also face audio quality challenges and limitations. What’s important is whether voices are intelligible in these kinds of products—in playback and recording. Outdoor cameras are prone to artifacts such as wind noise, and as a result, speech becomes hard to understand. In the case of a doorbell camera, the audio quality is affected by the distance of the speaker to the device.  The loudness of the captured of voices is important, especially in comparison with the volume of the background (Signal to Noise Ratio) which may be affected either by the distance from the user to the device or by the environmental conditions (wind, rain, road noises, …).

Doorbell benchmark

With image and audio quality being such an integral part of home security systems, it seems natural that DXOMARK, an industry leader in image and audio quality evaluation, would develop a testing protocol for the different types of home surveillance systems.

For our first image quality tests on the home surveillance market, we tested 4 doorbell cameras: Google Nest Doorbell (battery), Google Nest Hello (wired), Ring Video Doorbell 4, and the Arlo Essential Video Doorbell. All were battery-powered except for the Google Nest Hello. Of the four doorbell cameras, only the Google Nest Doorbell provided a vertical field of view instead of a horizontal one. A vertical field of view provides a head-to-toe view of the person in front of the camera as well as a view of the ground, where delivered packages are likely to be left.

The introductory protocol tested the doorbell cameras for all relevant attributes of image quality. It pays particular attention to the attributes of exposure, detail preservation, and artifacts, as they are identified as the most important for the doorbell use cases. Installation, accurate detection, and audio were not evaluated.  Videos were downloaded for analysis and were not evaluated via smartphone applications.

As the videos and illustrations will show, the doorbell cameras were tested in laboratory and real-life conditions. DXOMARK’s methodology combines both objective and perceptual evaluations to arrive at an overall assessment.

Let’s take a closer look at what DXOMARK’s laboratory and perceptual testing revealed about the image quality of doorbell cameras in three conditions: Daylight, Backlit, and Night.

Daylight use cases

When it comes to exposure in daylight conditions, with sunlight on the subject, faces were recognizable and accurately exposed in the images taken with the  Google, Ring cameras. While faces were recognizable in the Arlo camera, they were overexposed. The  Arlo tended to clip the face in very sunny conditions, because of high target exposure.

All cameras showed an acceptable level of detail in their images, but it depended on the subject’s distance from the camera. Identifying the individual was possible for close targets, but when the subject stood  2 meters or more away from the camera, facial details were almost completely lost. As the video shows, we have subjects approaching the camera from a distance in order to see how the level of details changes.

Some compression artifacts, such as blocking, can appear in the scene, especially with the subject’s movements. This happens when the video codec can’t keep up with the changing pixel information induced by the subject’s movement. More motion means more pixels have to change from one frame to another, and the more likely blocking could occur.

Arlo Video Doorbell (battery)

Ring Video Doorbell 4 (battery)

Google Nest Hello (wired)
Target Exposure Graph EV0. The EV (exposure value) reflects the dynamic of the lab scene, as the difference between the light on the realistic mannequin and the light produced by the LED panel in the scene.
The two graphs show the results from our lab tests for target exposure and texture. Target exposure (top)  was accurate for all four cameras, with the Arlo showing the highest level of exposure, which is too high. In the graph above, details are low, but acceptable, for the Ring and Google Nest Hello Wired.
Arlo Essential (battery), 1000Lux D65 EV0)
Google Nest Doorbell (battery), 1000Lux D65 EV0
Google Nest Hello Doorbell (wired), 1000Lux D65 EV0
Ring Video Doorbell 4 (battery), 1000Lux D65 EV0

Challenging backlit conditions

Backlit conditions create a strong dynamic in the scene, which is the case if the camera is installed on a porch, or covered entrance, for example. All cameras struggle to give an accurate target exposure on the face, and recognition is not assured. In our tests, the battery-powered  Google Nest managed to keep quite a high level of details on faces in this type of condition, while the others had low details.
The following videos of an HDR backlit condition illustrate a common use case of a daytime package delivery, and the doorbell camera is placed in a covered entrance.

Arlo Essential, underexposes

Google Nest Hello (wired)

Google Nest (battery)
In lab conditions, we can see as well that the target exposure measurements are lower in EV4 conditions than in EV0 conditions for all cameras.
In a backlit condition, the Arlo Essential, however,  overexposes the image. The Nest doorbell battery starts to have slightly low target exposure.
Arlo Essential
Google Nest (battery)
Ring Video Doorbell 4
 The Arlo Essential has a very limited dynamic range (low entropy). Other doorbells are similarly managing to keep some details in bright parts (The Google Nest battery and Ring are superimposed on one another  on this graph because the results were the same. )

Night use cases

The doorbell cameras’ behavior was interesting when it came to night use cases.  When light levels decrease or are really low, the cameras switch to night vision, or infrared mode, to better see.
The cameras had different thresholds at which they would activate night vision. The Ring Doorbell 4 switches to night vision at a lower lux level than the other doorbells. In very low light conditions where there was just one light on the model, most doorbells struggled to not overexpose the face. As the videos below show, in some cases, the subject’s face was so overexposed that it was impossible to identify the person.
In the following examples of a night scene, the cameras recorded the scene in IR. A small light simulated a lamp over the front door.

Arlo Essential, low detail preservation

Google Nest Hello, accurate exposure on the subject, but low details

Ring Video Doorbell 4, target exposure is the background and not the face.
.
In the laboratory examples below of infrared capabilities, we can see how the difference among the cameras when it comes to preserving details. 
Arlo Essential Video Doorbell
Google Nest (battery)
Google Nest Hello (wired), lowest target exposure, but better details
Ring Video Doorbell 4

Conclusion

The DXOMARK doorbell camera protocol evaluates exposure, detail preservation, artifacts, and color, using our latest innovations in image-quality testing: ultra-realistic mannequins and laboratory illuminants that allow us to repeatably and accurately reproduce the conditions in which these cameras are used.

Based on our evaluations of one wired doorbell camera and three battery-powered ones,  the Google Nest Hello (wired), gave better image results overall, providing high enough quality to identify people in most conditions, even at night. Overall, battery-powered doorbells held up well in comparison, but the Ring and Google Nest (battery) struggled when it came to low light and night conditions. Still, all doorbell cameras are challenged to some extent to provide correct exposure and facial details when confronted with  HDR conditions such as sunsets or covered entrances.

We didn’t talk about color in this initial article, since accurate and pleasant colors on surveillance cameras are nice to have but are not a mandatory feature. In low-light conditions, the cameras switched to infrared mode, where color information is lost. But in the daylight use cases we tested, all cameras showed some sort of color casts in daylight use cases in both perceptual and lab testing.

Results of the DXOMARK’s doorbell camera benchmark, a protocol of home security cameras.
DXOMARK is evaluating other types of home security cameras, as well, such as outdoor and indoor cameras. More articles about how those cameras performed are coming soon.

DXOMARK invites our readership (you) to post comments on the articles on this website. Read more about our Comment Policy.