Headphone Testing article: A Visit to Reviewed.com
Author: Zarina Bhimani. Reprinted from the June 2016 issue of Voice Coil.
In this article we discuss headphone testing using SoundCheck at independent review site, Reviewed.com. Includes description of test lab, measurement methods and more.
Full Article in VoiceCoil Magazine (PDF)
Article text
A Visit to Reviewed.com
Reviewed.com, part of the USA Today network, carries out quantitative reviews on a wide range of products including appliances, headphones, cameras, televisions and more. Since the beginning, its reviews have been built on the principle of using standardized scientific testing procedures to examine the performance of products, and a proprietary scoring method to ensure a level playing field amongst all manufacturers. Recently, I met with senior scientist Julia MacDougall, toured the facility, and learned about its headphone test methods. I also received a demonstration of the company’s recently upgraded SoundCheck system.
The large brick building in Central Square, Cambridge, MA, is in a part of town renowned for its young startup culture and unconventional work environments, so it’s no surprise to see a ping pong table next to the large, glass-walled conference room. However, once you get beyond the main lobby, there is a labyrinth of test labs, each designed for testing a specific product. A room dedicated to camera testing features various test pictures on the walls, as well as 3-D models with many moving and rotating parts to evaluate the camera’s capture of movement. Another lab was filled with massive flat-screen televisions that were being tested for display performance, color measurement, luminance, contrast, and more. Perhaps the most impressive was the appliance lab, where staff get to do their own laundry while they work (in the interests of testing the washers) to scientifically evaluate the performance of the washing machines. Dishwashers, dryers, microwaves, ovens, and refrigerators are also tested there. Then, testers retreat to the floor above to write up product reviews for the company website.
The Audio Test Lab
The area that interested me the most was the smallest test area—the audio lab. Headphones are small and the test equipment is compact so a large room is unnecessary. Reviewed.com has been using Listen Inc.’s SoundCheck software since it first started looking for an objective way to test audio products back in 2007. Back then, SoundCheck was used for measuring mobile phones—smartphones were in their infancy and Reviewed.com was the first review website to measure the sound quality of a wide range of phones.
Reviewed.com’s audio test focus has changed over the years. Emphasis is placed on products customers want to know more about before buying, and the review focuses on the product most important to the customer. Since the smartphone market has matured and customer choice is driven by brand loyalty and the ecosystem over audio performance, Reviewed has moved on to testing other products. Headphones are one of the largest and fastest growing segments of the consumer electronics industry (in part, driven by the smartphone revolution), so now the audio test lab is primarily focused on headphones.
The audio test lab is a small, climate-controlled room on the ground floor of the building. It contains a computer with the SoundCheck system, a Brüel & Kjær Sound & Vibration Measurement A/S Head-and-Torso simulator (HATS), an amplifier to drive the headphones, speakers used for noise cancellation tests, and two SoundConnect microphone power supplies to power the ears of the HATS. The walls and the door are entirely covered in acoustical foam to acoustically isolate the room and minimize reflections. Other precautions have also been taken to ensure accurate testing. The HATS sits on a rigid rack mounted to the wall to prevent vibration. Initially, it was on a desk above the computer but the vibration from the computer fan interfered with the measurements. The Reviewed.com offices also have a more unusual noise issue to contend with. Due to its proximity to the local subway line, there is a low-frequency rumble every 5 minutes or so when a train goes by! Measurements are stopped or repeated if the train is not heard until too late.
Headphone Testing
Reviewed.com has been testing wired headphones for several years now, and has recently updated its setup to test the audio performance of wireless headphones for the same performance standards.
Figure 1 shows the test setup for a typical wired headphone. In-ear, over-the-ear, and on-the-ear headphones are tested using the same setup. The HATS is, as the name indicates, a device that replicates the acoustic behavior of a human head and body. The pinna is constructed to accurately replicate the average human ear, and behind the pinna is a very accurate measurement microphone representing the human eardrum. These microphones require 200 V polarization and are powered using SoundConnect microphone power supplies from Listen. When headphones are placed on the HATS, they must be placed as accurately as possible since small variations in fit can affect the test results. For this reason, four measurements are made on each headphone, and the best one is used for evaluation.
The SoundCheck software generates the test signal and receives and processes the recorded response. The test signal, a stepped sine sweep, is transmitted to the headphones via the audio interface headphone output of the amplifier. The sound is recorded by the microphones in the HATS and returned to the computer via the audio interface so it can be analyzed by the SoundCheck software.
Reviewed.com carries out six specific tests on every headphone: frequency response, distortion, tracking, leakage, isolation, and sound pressure level (SPL). The first three of these are the most important as they are the ones that have the greatest influence on the perceived sound quality.
Frequency response, distortion, and tracking are simultaneously measured using a stepped sine sweep from 20 Hz to 20 kHz. Figure 2 depicts a screenshot that shows the output from this test.
All measurements are then compared to standard curves to enable Reviewed.com to apply a numerical value to the data, which makes up the overall score. This enables unbiased and simple comparison between headphones, and these numerical values contribute to the device’s total score.
Audio Testing
Frequency Response – For measuring frequency response, recorded sound is compared to the original sound file to determine how the headphones have altered the sound. For consumer headphones, the frequency response is compared to the ISO 226:2003 equal loudness curve standard, which is the curve at which human ears hear notes at the same loudness. For studio headphones, limits are set at ±5 dB (SPL) against the response curve of the headphones. This means these headphones are not scored on the exact shape of the curve but rather how much the headphones deviate from these limits.
An objective measurement of distortion is obtained by measuring the total harmonic distortion (THD), a measurement of the distortion at every harmonic in addition to the fundamental. This is plotted and compared to a proprietary empirical data curve which represents Reviewed.com’s acceptable threshold, based on more than six years of headphone test results. Any measurements above this line have a negative effect on the score.
Tracking is a measurement of how the channels sound compared to each other. Ideally, the left and the right ears should sound the same. Both are simultaneously measured across a range of frequencies from 20 Hz to 20 kHz. Any difference of more than ±2 dB affects the score.
Other performance tests measure leakage, isolation, and maximum SPL. Leakage is an indication of how much sound escapes from the headphones. This is measured using a sound level meter placed precisely 6” from the ear. Initially, the ambient noise of the room is measured, pink noise is played back through the headphones at 90 dB, and the volume measured. A simple subtraction of the background noise from the measured noise offers a numeric leakage value that contributes to the overall score.
Isolation, the ability of the headphones to eliminate outside noise, is also measured using SoundCheck. The setup enables both passive isolation (i.e., isolation due to the mechanical structure of the headphone) and active isolation (i.e., the noise cancelling functionality) to be measured. Figure 3 shows the measurement when pink noise is played at 90 dB without the headphones on. Its is level measured across the frequency spectrum. The headphones are placed on the HATS, and the noise played again and measured so that the attenuation is calculated. In noise cancelling headphones, a third measurement is made with the active noise cancellation turned on. By subtracting active and passive noise cancellation curves from the unoccluded curve, the isolation can be numerically quantified. This is compared to the average values of hundreds of pairs of headphones to calculate their noise isolation score. Last, SPL is measured. In this test, the volume of the stimulus signal is increased and the distortion analyzed until the peak THD reaches 3%, or the level of sound reaches 120 dB. There is no need to test higher than 120 dB as headphones are not intended to be played at that volume due to the risk of permanent hearing damage.
Wireless Headphone Testing
Recently, more and more headphones are being used in wireless mode, so Reviewed.com decided that it was important to test them under such conditions. (Until recently, wireless headphones were tested in wired mode.) The addition of a Bluetooth interface, the BQC-4148, to the test setup now enables wireless headphones to be tested exactly the same way as their wired counterparts (see Figure 4).
Instead of routing the test signal from the computer to the headphone via a headphone amplifier, it is routed via the BQC1448 Bluetooth interface. This small device connects to the computer via a USB, and is controlled via SoundCheck, where parameters such as the Bluetooth Protocol and transmitter power are set. It is paired with the headphones under test, and the signal is transmitted via the interface directly to the headphone. The recording and analysis side of the setup is exactly as before—the signal is transferred from the microphones within HATS to the computer via an audio interface and analyzed.
With this setup, wireless headphones are measured to exactly the same standards as wired headphones, so the score they receive is truly representative of the way they are commonly used. It is even possible to compare wired and wireless performances of the headphone by making the same measurement with both the conventional and Bluetooth setup.
In addition to the audio tests, the headphones are evaluated for comfort, control, and functionality. Specialty headphones (e.g., sports headphones) undergo additional testing, such as being worn on a long run to test for a secure fit in active conditions.
The Future
Mobile audio technology is evolving extremely fast. In the past 10 years, we have seen the smartphone revolution, the explosion of the headphone industry, the introduction of wireless headphones (in fact, wireless everything), the return of high-resolution audio, voice activated audio, and more. In addition, wireless homes are generating new audio opportunities (e.g., light bulbs that also function as wireless speakers). Reviewed.com will follow these trends, testing the hottest consumer products, particularly those where performance is a big differentiator.
The SoundCheck system is inherently flexible for testing any device, as it can support up to 64 channels of audio and there a range of interfaces enables Bluetooth, USB, MicroElectrical-Mechanical System (MEMS) microphones, and more to be tested. High-end audio interfaces permit accurate testing of high-resolution audio and the ability to custom-program virtually any test means the possibilities are endless.
This makes it a valuable tool for a constantly evolving product review site such as Reviewed.com, because the flexibility and forward compatibility of the SoundCheck system ensures that it will be capable of testing any audio device. This has already been demonstrated with the recent system upgrade to add the Bluetooth interface and update the software. As MacDougall, so succinctly explained, “We are very excited to use SoundCheck and the BQC4148 Bluetooth interface to broaden our headphone tests to include wireless headphones—it enables us to offer better, more relevant data to our readers.”
Curious to learn more ? Check out our main page about headphone testing.
Learn about the latest in headphone test systems and methods from design to EOL Test. Industry experts Steve Temme (Listen, Inc.) and Peter Wulf-Andersen (G.R.A.S.), as well as guest speaker Chris Struck will discuss test equipment (ear couplers, test fixtures, test software and hardware) and demonstrate practical test setups for both R&D and QC headphone testing. It will cover in-ear monitors, Bluetooth, lightning/USB, noise-cancelling and high resolution headphones, in addition to conventional analog headphones. You can even measure your own headphone using our equipment in the hands-on session. Please see the full agenda below.
