A Visit to Reviewed.com
How an independent review website uses SoundCheck to ensure fair and quantitative product assessment
About Reviewed.com
Reviewed.com, part of the USA Today network, carries out quantitative reviews on a wide range of products including appliances, headphones, cameras, televisions and more. Since the beginning, their reviews have been built on the principle of using standardized scientific testing procedures to examine the performance of products, and a proprietary scoring method to ensure a level playing field amongst all manufacturers. Recently, I met with senior scientist Julia MacDougall, and received a tour of the facility and some insight into their headphone test methods, as well as a demonstration of their recently upgraded SoundCheck system.
The large brick building in Central Square, Cambridge, is in a part of town renowned for its young start up culture and unconventional work environments, so it’s no surprise to see a ping pong table next to the large, glass-walled conference room. However, once you get beyond the main lobby it is a labyrinth of test labs, each designed for testing a specific product. A room dedicated to camera testing features various test pictures on the walls, as well as 3d models with many moving and rotating parts to evaluate the camera’s capture of movement. Another lab was filled with massive flat screen televisions that were being tested for display performance, color measurement, luminance, contrast and more. Perhaps the most impressive was the appliance lab, where staff get to do their laundry while they work (in the interests of testing the washers), as well as working their way through many loads of white towels and stain strips that are marked with red wine, chocolate, sweat and more to scientifically evaluate the performance of the washing machines. Dishwashers, dryers, microwaves and ovens are also tested here, and a dedicated temperature and humidity controlled room contains many refrigerators filled with ‘dummy food’, the temperature of which is continuously monitored. The floor above the test labs is where their testers retreat to write up product reviews for their website, away from the whirr of tumble driers, swishing of dishwashers and stepped sine waves from the audio test lab.
The Audio Test Lab
The area that interested me the most was the smallest test area – the audio lab. Headphones are small and the test equipment is also compact, so a large room is unnecessary. Reviewed.com has been using Listen’s SoundCheck software since they first started looking for an objective way to test audio products back in 2007. Back then SoundCheck was being used for measuring mobile phones – smartphones were in their infancy, the next ‘hot product’, and Reviewed.com was the first review website to measure sound quality of a wide range of phones.
Reviewed.com’s audio test focus has changed over the years. Emphasis is placed on products where customers want to know about the product before buying, and focuses on the aspects of those products that are most important to the customer. Since the smartphone market has now matured and customer choice is driven by brand loyalty and ecosystem over audio performance, Reviewed has moved on to testing other products. Headphones are one of the largest and fastest growing segments of the consumer electronics industry (in part driven by the smartphone revolution), so these are now a focus area, and the product that their audio lab is primarily used for.
The audio test lab is a small, climate-controlled room on the ground floor of the building. It contains a computer with the SoundCheck system, a Bruel and Kjaer Head and Torso simulator, an amplifier to drive the headphones and the speakers that are used for noise-cancellation tests, and 2 SoundConnect microphone power supplies to power the ears of the Head and Torso simulator. The walls and door are entirely covered in acoustical foam to acoustically isolate the room and minimize reflections. In addition to this, other precautions have been taken to ensure accurate testing. The Head and Torso simulator is on a rigid rack mounted to the wall to prevent vibration – initially it was on a desk above the computer, but vibration from the computer fan interfered with the measurements. The Reviewed offices also have a more unusual noise issue to contend with – due to their proximity to the local subway line, there is a low frequency rumble every 5 minutes or so as a train goes past! Measurements are stopped during this time, or repeated if the train is not heard until too late.
Headphone Testing
Reviewed.com has been testing wired headphones for several years now, and has recently updated their test setup to enable the audio performance of wireless headphones to also be tested to the same performance standards.
The diagram below shows the test setup for a typical wired headphone. In-ear, over-the-ear, and on-the-ear headphones are all tested using the same setup.
Fig 1: Test setup for standard wired headphone
The head and torso simulator is, as the name indicates, a device which replicates the acoustic behavior of a human head and body. The pinna are constructed to accurately replicate the average human ear, and behind them is a very accurate measurement microphone representing the human eardrum. These microphones require 200V polarization and are powered using SoundConnect microphone power supplies from Listen. When headphones are placed on the head and torso simulator, care must be taken to make sure they are accurately placed as small variations in fit can affect the test results. For this reason, 4 measurements are made on each headphone, and the best one used for evaluation.
The SoundCheck software both generates the test signal and receives and processes the recorded response. The test signal, a stepped sine sweep, is generated by the software and transmitted to the headphones via the audio interface headphone output of the amplifier. The sound is recorded by the microphones in the head and torso simulator, and returned to the computer via the audio interface so that it can be analyzed by the SoundCheck software.
Reviewed.com carries out 6 specific tests on every headphone: frequency response, distortion, tracking, leakage, isolation and sound pressure level (SPL). The first three of these are the most important as they are the ones that have the greatest influence on the perceived sound quality.
Frequency response, distortion and tracking are all measured simultaneously using a stepped sine sweep from 20Hz-20kHz. Below is a screenshot that shows the output from this test:
Fig 2: Final test result in SoundCheck. This data is exported and compared to data curves to calculate the headphone’s rating for each characteristic measured.
All measurements are then compared to standard curves to enable Reviewed.com to apply a numerical value the data, which makes up the overall score. This enables unbiased and simple comparison between headphones, and these numerical values contribute to the device’s total score.
Frequency Response
For measuring frequency response, recorded sound is compared to the original sound file to determine how the headphones have altered the sound. For consumer headphones, the frequency response is compared to the ISO 226:2003 equal loudness curve – the curve at which human ears hear notes at the same loudness. For studio headphones, limits are set of +/-5dB (SPL) against the response curve of the headphones. This means that these headphones are not scored on the exact shape of the curve but rather how much the headphones deviate from these limits.
Distortion
An objective measurement of distortion is obtained by measuring the Total Harmonic Distortion (THD), a measurement of the distortion at every harmonic in addition to the fundamental. This is plotted and compared to a proprietary empirical data curve which represents Reviewed.com’s acceptable threshold, based on 6+ years of headphone test results; any measurements above this line have a negative effect on the score.
Tracking
Tracking is a measurement of how the channels sound compared to each other – ideally left and right ear should sound the same. Both are simultaneously measured across a range of frequencies from 20Hz-20kHz, and any difference of more than +/-2dB affects the score.
Other performance characters measured are leakage, isolation and max SPL.
Leakage
Leakage is an indication of how much sound escapes from the headphones to annoy the person next to you. This is measured using a sound level meter placed precisely 6” from the ear. Initially the ambient noise of the room is measured, pink noise is played back through the headphones at 90dB, and the volume measured. A simple subtraction of the background noise from the measured noise offers a numeric leakage value that contributes to the overall score.
Isolation
Isolation, the ability of the headphone to eliminate outside noise, is also measured using SoundCheck. The setup enables both passive isolation (isolation due to the mechanical structure of the headphone) and active isolation (noise cancelling functionality) to be measured. In this setup (fig 3), pink noise is played at 90dB without the headphones on, and its level measured across the frequency spectrum. The headphones are placed on the HATS, and the noise played again and measured so that the attenuation is calculated. In noise cancelling headphones, a third measurement is made with the active noise cancellation turned on. By subtracting active and passive noise cancellation curves from the unoccluded curve, the isolation can be quantified numerically. This is compared to the average values of hundreds of pairs of headphones to calculate their noise isolation score.
Fig 3: Test setup for measuring active and passive noise cancellation
Lastly, SPL is measured. In this test, the volume of the stimulus signal is increased and the distortion analyzed until the peak THD reaches 3%, or the level of sound reaches 120dB. There is no need to test higher than 120dB as headphones are not intended to be played at that volume due to the risk of permanent hearing damage.
Wireless Headphone Testing
Recently, more and more headphones are being used in wireless mode, so Reviewed.com decided that it was important to test these under such conditions (up until recently, wireless headphones were tested in wired mode). The addition of a Bluetooth interface, the BQC-4148, to the test setup now enables wireless headphones to be tested exactly the same way as their wired counterparts (fig. 4).
Instead of routing the test signal from the computer to the headphone via a headphone amplifier, it instead is routed via the BQC1448 Bluetooth interface. This small device connects to the computer via a USB, and is controlled via SoundCheck, where parameters such as the Bluetooth Protocol and transmitter power are set. It is paired with the headphones under test, and the signal is then transmitted via the interface directly to the headphone. The recording and analysis side of the setup is exactly as before – the signal is transferred from the microphones within HATS to the computer via an audio interface and analyzed.
Fig 4: Test setup for measuring wireless (Bluetooth) headphones
With this setup, wireless headphones are measured to exactly the same standards as wired headphones, so that the score they receive is truly representative of the way they are commonly used. It is even possible to compare wired and wireless performance of the headphone by making the same measurement with both the conventional and Bluetooth setup.
In addition to the audio tests, the headphones are evaluated for comfort, control, and functionality. Specialist headphones, for example sports headphones undergo additional testing, such as being worn on a long run to test for a secure fit in active conditions.
The Future
Mobile audio technology is evolving extremely fast. In the past 10 years we have seen the smartphone revolution, the explosion of the headphone industry, the introduction of wireless headphones (in fact, wireless everything), the return of high-resolution audio, voice activated audio and more. In addition, wireless homes are generating new audio opportunities, for example light bulbs that also function as wireless speakers. Reviewed.com will follow these trends, testing the hottest consumer products, particularly those where performance is a big differentiator.
The SoundCheck system is inherently flexible for testing any device, as it can support up to 64 channels of audio and there a range of interfaces enables Bluetooth, USB, MEMs microphones, etc. to be tested. High end audio interfaces permit accurate testing of high-res audio, and the ability to custom-program virtually any test means the possibilities are endless. This makes it a very valuable tool for a constantly evolving product review site such has Reviewed.com, as the flexibility and forwards compatibility of the SoundCheck system ensures that it will be capable of testing any audio device. This has already been demonstrated with the recent upgrade of the system to add the Bluetooth interface and update the software to the latest version. As Julia MacDougall, Reviewed.com’s senior scientist, so succinctly explained, “We are very excited to use SoundCheck and the BQC4148 Bluetooth interface to broaden our headphone tests to include wireless headphones – it enables us to offer better, more relevant data to our readers.”
