Helping Consumers Make the Right Purchase: DXOMARK Assesses Smartphone Image and Audio Quality Using Rigorous Test Protocols

TL; DR: DXOMARK is an independent source for test results and reviews of mobile cameras, lenses, and digital cameras, along with smartphone audio and playback. The organization, powered by a dedicated team of imaging and audio engineers, assesses devices using industry-grade lab tools and a rigorous scientific testing protocol. DXOMARK continues to introduce enhancements in step with rapidly evolving mobile technology, such as a selfie test suite and tests for specialized photography capabilities like Zoom and Bokeh.

We may refer to the devices we keep in our pockets as smartphones, but when it comes down to it, the ability to make calls is often overshadowed by more significant features.

When Apple released the iPhone 11 Pro in September 2019, for example, it was immediately clear that its triple-lens camera system, featuring an ultra-wide-angle lens, was the star of the show.

“The smartphone user of today pays a lot of attention to the camera when making purchases,” said Anwesha Ghosh, Marketing Project Manager at DXOMARK. “Modern consumers consider it one of the most important components.”

Because customers can’t always trust advertising messages or incentivized online reviews to shape their buying decisions, DXOMARK has provided independent benchmarks gauging the quality of cameras and lenses since 2008.

To keep up with rapid advancements in smartphone technology, the organization introduced a mobile image protocol in 2012, followed by a similar audio protocol in 2019. Further enhancements, such as a selfie test suite and benchmarks for increasingly common features like Zoom and Bokeh, are frequently introduced in step with demand.

Over the years, DXOMARK has earned a reputation for assessing devices using industry-grade lab equipment and scientifically sound, repeatable testing measures. A growing team of more than 60 image quality engineers and technicians perform all tests and data analytics, drawing on their unique skills in electronics, camera mechanics, image science, optics, chip design, sensors, and coding. Ultimately, the organization’s goal is to educate consumers on the quality of devices via a quantifiable and objective approach to product comparison.

Anwesha told us that DXOMARK was formerly part of DXO Labs, which was founded in 2003 to provide a variety of consumer software and hardware products for photo editing. One such product was DXO One, an iPhone accessory designed to boost image quality.

“When we were developing DXO One and other products, we needed a process to evaluate image quality for our internal purposes,” Anwesha said. “There was no such solution available on the market, so we had to develop a method on our own, which was a complex process.”

After testing devices internally, the team realized the information would prove valuable to the general consumer, and the company began to release test results for various Digital Still Cameras (DSCs) and single-lens reflex (DSLR) cameras. Next, it added lenses to the mix.

As the industry evolved, first with the iPhone and then with various other smartphones, DXOMARK launched a protocol for smartphone cameras before becoming an independent company spun off from DXO Labs in 2017.

“Technology changes every day — and changes happen very quickly,” Anwesha said. “If you look back at the earliest smartphones, they were nowhere close to DSLRs in terms of quality. Today, we can see that the latest generations of high-end smartphones, a lot of them are rapidly approaching that value.”

The biggest constraint on smartphone camera performance is the form factor, as the market demands thin phones. Anwesha said the addition of multiple cameras has made a range of advanced features possible.

As smartphone technology continues to become more sophisticated, DXOMARK is extending its mobile protocols to evaluate features such as wide-angle lens capabilities and night mode for low-light photography.

The organization also tests front-facing cameras to generate a DXOMARK Selfie score made up of sub-scores evaluating exposure, contrast, color, focus, texture, noise, artifacts, flash, and bokeh.

“As trivial as it sounds, there’s a pretty sophisticated technology that enables a selfie to be taken, and more and more people depend on their smartphones for conveniences like that,” Anwesha said.

Adherence to rigorous scientific standards is at the heart of everything DXOMARK does. Anwesha told us that the organization places a high priority on maintaining neutrality.

“When you say something is scientific, it needs to follow certain principles,” she said. “The tasks we perform need to be repeatable and robust in the sense that every time we test a phone, no matter what phone, it has to be tested in the exact same manner.”

The testing, performed by experts in the field, is also incredibly comprehensive — so much so that when a new person joins the company, they are trained for six months before they are assigned to a project.

“When you look at the DXOMARK score for a smartphone, it is composed of a photo and a video score, and each photo score is compared to some seven or eight other scores,” Anwesha said. “And each of those scores is composed of about five to 10 different measures. They’re also tested under various lighting conditions.”

The organization also updates its test protocol on a regular basis to reflect the technology that is currently on the market. This helps ensure readers are receiving relevant information on the latest products, fostering consumer trust and assuring that tech buyers get the most substantial return on their investments.

“Our main mission is to educate consumers, and we do that objectively by remaining extremely factual,” Anwesha said. “We never, ever guess or comment about a product until we test it with rigorous protocols. We want to educate in the right way.”

One of the company’s most recent innovations is its DXOMARK audio test protocol, which was announced on October 10, 2019. The protocol, intended to provide reliable, neutral, and comprehensive data on smartphone audio recordings and output, was developed to mirror the ways consumers use their smartphones to consume audio content.

Before launching the new protocol, DXOMARK built a dedicated audio team with vast experience in electronic engineering, acoustic industrial design, and software engineering. Anwesha said the group then spent more than a year acquiring equipment, building a state of the art audio lab, and developing a distinct testing method.

“Internally, this was an investment, requiring us to use our time and resources to deliver expertise that we previously did not have,” she said.

The company’s first ranking tested the audio capabilities of seven smartphones: Apple iPhone 11 Pro Max, Apple iPhone XS Max, Samsung Galaxy Note 10+, Samsung Galaxy S10+, Honor 20 Pro, Huawei Mate 20 X and Sony Xperia 1.

The top spot went to Huawei Mate 20 X, which earned 75 points, followed by the iPhone XS Max and iPhone 11 Pro Max at 74 and 71 points, respectively. Samsung Galaxy Note 10+ and S10+ were ranked next, with Honor 20 Pro and Sony Xperia 1 earning the lowest rankings.

“For us to stay ahead of the curve, we had to adapt our testing protocol to make sure when the technology changes, we are able to measure it and determine how the different kinds of new technology compare,” Anwesha said. “We have a pretty big lineup of things coming up next year in terms of testing various other capabilities of devices.”