Posts

Testing Audio Performance of Hearables

/in /by
Picture of AES paper on testing hearables

Testing Hearables AES Paper

Testing hearables, or smart headphones, is challenging. They have various interfaces ranging from hardwired to wireless and often contain signal processing on both the record and playback side. This means that their characteristics change according to ‘real world’ conditions such as their physical environment and background noises. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback or even operation as a telephone headset or hearing aid. In this AES paper, we discuss how to implement basic acoustic tests as well as the more complex real-world tests, techniques, standards, and equipment that are necessary.

Authors: Steve Temme, Listen, Inc.
Presented at AES Headphone Conference 2019, San Francisco, CA.

Full Paper
Poster Presentation

 

Abstract & introduction for “Testing Audio Performance of Hearables”

Abstract for “Testing Hearables”
Smart headphones or “hearables” are designed not only to playback music but to enhance communications in the presence of background noise and in some cases, even compensate for hearing loss. They may also provide voice recognition, medical monitoring, fitness tracking, real-time translation and even augmented reality (AR). They contain complex signal processing and their characteristics change according to their smartphone application and ‘real world’ conditions of their actual environment, including background noises and playback levels. This paper
focuses on how to measure their audio performance under the many various real-world conditions they are used in.

Introduction for “Testing Hearables”
Hearables are notoriously challenging to test. They have various interfaces ranging from hardwired to wireless (e.g. Bluetooth) and may contain much signal processing, both on the record side (e.g. beamforming, background noise filtering, voice activity detection, and on the playback side (e.g. loudness, compression, equalization, and active noise cancellation). This means that their characteristics change according to ‘real world’ conditions such as their physical environment and background noises. Some even have wake word detection, e.g. ‘Hey Siri’. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback or even operation as a telephone headset or hearing aid. Due to their complex non-linear use cases, these devices often need to be tested at different levels and in different environmental conditions, for example with background noise, different signals etc. Although, there are currently no standards for testing smart devices such as hearables, we can borrow principles and test configurations from many other audio devices and use existing standards such as; IEC for headphones [1], IEEE for headsets [2], ETSI for background noise [3], TIA/ITU for telephone test [4] and ANSI for hearing aids standards [5].

Flexibility of the test system and experience with testing a wide range of acoustic devices is critical to enable a device to be completely characterized. This paper discusses how to implement basic acoustic tests and some of the more complex real-world tests along with the techniques and standards that may be used. Test system requirements for measuring voice
enabled hearables will also be discussed.

Full Paper

More about Headphone & Hearables Testing

European Tour: Headphone & Headset Measurements

/in , /by

The challenges of testing today’s devices

Presented by:

listen_gras_logos2

 

 

headphonesLearn about the latest in headphone test systems and methods from design to EOL Test. Industry experts Steve Temme (Listen, Inc.) and Peter Wulf-Andersen (G.R.A.S.), as well as guest speaker Chris Struck will discuss test equipment (ear couplers, test fixtures, test software and hardware) and demonstrate practical test setups for both R&D and QC headphone testing. It will cover in-ear monitors, Bluetooth, lightning/USB, noise-cancelling and high resolution headphones, in addition to conventional analog headphones. You can even measure your own headphone using our equipment in the hands-on session. Please see the full agenda below.

There is no charge for this one-day seminar and lunch will be provided. Space is limited, so please RSVP today.

Locations and Dates:

UK: Monday May 15th, Sharnbrook Hotel, Bedford, England

Denmark: Thursday May 18th, G.R.A.S. Headquarters, Holte (near Copenhagen), Denmark

Germany: Friday May 19th, Maritim Airport Hotel, Hannover, Germany

Print

 

See what people are saying about this event (we recently presented it in the US)

Agenda (9am-4pm)

  • Introduction
  • Overview of Headphone Test Systems and tools
    • Traditional tools vs. modern consumer demands
      • Introduction to head and ear related terminology and definitions.
      • Historical perspective and background for the tools and standards used today.
      • Limitations vs. Possibilities in term of the measurement capabilities. How does that relate to the modern challenges with Bluetooth and ANC related testing tasks?
    • Tools for Next Generation Headphone Testing
      • Solutions to some the challenges addressed above.
      • Examples based on products and measurements.
    • What about frequencies above 20kHz? Science or marketing? Measurement Consistency – Development vs. Production
      • Progress in measurements from Head and Torso Simulator (KEMAR) to Production Test Fixture (45CC – flat plate). How the different components affect the measurements:
    • Software and hardware for headphone testing including typical headphone measurements and test setups
  • Practical examples of headphone measurements on.
    • Analog Headphones
    • Bluetooth Headsets
    • ANC Headphones
    • Lightning/USB Connected Headphones (using triggered recordings)
    • Hi-Res Headphones
    • In-ear monitors
  • Measurements including:
    • Frequency Response relative to a target response
    • Left/Right tracking
    • Impedance
    • Noise attenuation
    • Distortion including Non-Coherent Distortion
    • Standards including Max SPL EN-50332 & IEC-60268-7
  • Headphone Design (Guest Speaker Chris Struck, CJS Labs)
    • Desired headphone response
    • Headphone measurement standards
  • Hands-on testing: Bring your own device and test it using one of our measurement setups

Print

Reserve your space

Headphone Testing with SoundCheck ONE

/in /by

This SoundCheck ONE template sequence contains all the essential steps for basic headphone measurements using SoundCheck ONE and AudioConnectTM. The sequence can be easily customized and saved for specific products by turning individual measurements on and off, and by adjusting settings within each sequence step such as stimulus range and level, tolerance limits, graphical displays, and data saving.
Please note that sequences in SoundCheck ONE cannot have steps added/removed or the layout modified – the full version of SoundCheck is required for this capability.

More

The Correlation Between Distortion Audibility and Listener Preference in Headphones

/in /by
Picture of paper on listener preference & distortion audibility in headphones

Listener Preference & Distortion Audibility in Headphones

The correlation between listener preference and distortion audibility is investigated in this AES paper from Steve Temme, Dr. Sean Olive et al. Five popular headphones with varying degrees of distortion were selected and equalized to the same frequency response. Trained listeners compared them subjectively using music as the test signal, and the distortion of each headphone was measured objectively using SoundCheck. The correlation between subjective listener preference and objective distortion measurement is evaluated and discussed.

Authors: Steve Temme, Sean E. Olive*, Steve Tatarunis, Todd Welti*, and Elisabeth McMullin*            *Harman International
Presented at the 137th AES Conference, Los Angeles 2014

Full Paper

 

 

Listener Preference & Distortion Paper Abstract & Introduction

Abstract
It is well-known that the frequency response of loudspeakers and headphones has a dramatic impact on sound quality and listener preference, but what role does distortion have on perceived sound quality? To answer this question, five popular headphones with varying degrees of distortion were selected and equalized to the same frequency response. Trained listeners compared them subjectively using music as the test signal, and the distortion of each headphone was measured objectively using a well-known commercial audio test system. The correlation between subjective listener preference and objective distortion measurement is discussed.

Introduction
There has been much research published on how a loudspeaker’s linear performance, e.g. frequency, time and directional responses, affects perceived sound quality. However, there is little research published on how non-linear distortion affects perceived sound quality. In recent years, the increasing availability and affordability of high quality headphones and personal digital music
players e.g. MP3 players, has brought high quality music playback to the masses. The transducer performance is critical to listener enjoyment and Dr. Olive and others have presented research on what they believe the target frequency response of the headphone should be for optimum sound quality [1]. The attempt of this research is to determine what level and what kind of distortion is audible and how it affects the perceived sound quality.

Five different pairs of good quality over-the-ear headphones with varying levels of distortion were objectively measured and subjectively rated for their perceived sound quality. First, each headphone was equalized to the same target frequency response. Several different kinds of distortion metrics including harmonic, intermodulation, and non-coherent distortion, were measured for each headphone. A listening test was then conducted where the five headphones were rated by eight trained listeners based on preference and distortion using four short musical excerpts. The program material was selected for wide dynamic and frequency ranges to excite mechanisms in the headphone transducers that would cause distortion.

The different headphones were presented virtually to listeners via binaural recordings of the headphones reproduced through a calibrated low-distortion reference headphone, Stax SR-009. This virtual headphone test method minimized headphone leakage effects, and removed the influence of non-auditory biases (brand, price, visual appearance, comfort, etc.) from listeners’ judgment of sound quality. In this paper, correlations between subjective and objective ratings of distortion are examined (as was done previously [2]) in an attempt to develop an objective metric for measuring distortion audibility in headphones and other loudspeakers. This could possibly be extended to other types of audio devices such as amplifiers.

 

More about Headphone Testing using SoundCheck

A visit to Reviewed.com

/in , /by

Headphone Test using SoundCheckAbout Reviewed.com

Reviewed.com, part of the USA Today network, carries out quantitative reviews on a wide range of products including appliances, headphones, cameras, televisions and more. Since the beginning, their reviews have been built on the principle of using standardized scientific testing procedures to examine the performance of products, and a proprietary scoring method to ensure a level playing field amongst all manufacturers. Recently, I met with senior scientist Julia MacDougall, and received a tour of the facility and some insight into their headphone test methods, as well as a demonstration of their recently upgraded SoundCheck system.

The large brick building in Central Square, Cambridge, is in a part of town renowned for its young start up culture and unconventional work environments, so it’s no surprise to see a ping pong table next to the large, glass-walled conference room. However, once you get beyond the main lobby it is a labyrinth of test labs, each designed for testing a specific product. A room dedicated to camera testing features various test pictures on the walls, as well as 3d models with many moving and rotating parts to evaluate the camera’s capture of movement. Another lab was filled with massive flat screen televisions that were being tested for display performance, color measurement, luminance, contrast and more. Perhaps the most impressive was the appliance lab, where staff get to do their laundry while they work (in the interests of testing the washers), as well as working their way through many loads of white towels and stain strips that are marked with red wine, chocolate, sweat and more to scientifically evaluate the performance of the washing machines. Dishwashers, dryers, microwaves and ovens are also tested here, and a dedicated temperature and humidity controlled room contains many refrigerators filled with ‘dummy food’, the temperature of which is continuously monitored. The floor above the test labs is where their testers retreat to write up product reviews for their website, away from the whirr of tumble driers, swishing of dishwashers and stepped sine waves from the audio test lab.

 

The Audio Test Lab

The area that interested me the most was the smallest test area – the audio lab. Headphones are small and the test equipment is also compact, so a large room is unnecessary. Reviewed.com has been using Listen’s SoundCheck software since they first started looking for an objective way to test audio products back in 2007. Back then SoundCheck was being used for measuring mobile phones – smartphones were in their infancy, the next ‘hot product’, and Reviewed.com was the first review website to measure sound quality of a wide range of phones.

Full Story

IEC-60268-7 Headphone Sequences

/in /by

IEC-60268-7: Sound System Equipment – Part 7: Headphones and Earphones is an international standard intended to characterize the performance of headphones and earphones. The standard itself is a lengthy document, 9 Sections and 3 Annexes covering 46 printed pages. These SoundCheck sequences focus on the electro-acoustic tests which are detailed in Section 8 “Characteristics to be specified and their method of measurement”.

Five separate sequences are provided, each designed to measure specific characteristics. This approach provides the user with the flexibility to measure all or some of the characteristics of their headphone.

More

Advances in Impedance Measurement of Loudspeakers and Headphones

/in /by

Impedance measurement is often the sole electrical measurement in a battery of QC tests on loudspeakers and headphones. Two test methods are commonly used, single channel and dual channel. Dual Channel measurement offers greater accuracy as both the voltage across the speaker (or headphone) and the reference resistor are measured to calculate the impedance. Single Channel measurement methods are more commonly used on the production line because they only require one channel of a stereo soundcard, which leaves the other free for simultaneous acoustic
tests. They are less accurate, however, due to the test methods making assumptions of constant voltage or constant current. In this paper we discuss a novel electrical circuit that offers similar impedance measurement accuracy compared to complex dual channel measurement methods but using just one channel. This is expected to become popular for high throughput production line measurements where only one channel is available as the second channel of the typical soundcard is being used for simultaneous acoustic tests.

Authors: Steve Temme and Tony Scott
Presented at the 135th AES Conference, New York 2013

Full Paper

Measurement of Harmonic Distortion Audibility Using A Simplified Psychoacoustic Model – Updated

/in /by

A perceptual method is proposed for measuring harmonic distortion audibility. This method is similar to the CLEAR (Cepstral Loudness Enhanced Algorithm for Rub & buzz) algorithm previously proposed by the authors as a means of detecting audible Rub & Buzz which is an extreme type of distortion[1,2]. Both methods are based on the Perceptual Evaluation of Audio Quality (PEAQ) standard[3]. In the present work, in order to estimate the audibility of regular harmonic distortion, additional psychoacoustic variables are added to the CLEAR algorithm. These variables are then combined using an artificial neural network approach to derive a metric that is indicative of the overall audible harmonic distortion. Experimental results on headphones are presented to justify the accuracy of the model.

Authors: Steve Temme, Pascal Brunet and Parastoo Qarabaqi
Presented at the 51st AES Conference, Helsinki, Finland, 2013

Full Paper

Measurement of Harmonic Distortion Audibility Using A Simplified Psychoacoustic Model

/in /by

A perceptual method is proposed for measuring harmonic distortion audibility. This method is similar to the CLEAR (Cepstral Loudness Enhanced Algorithm for Rub & buzz) algorithm previously proposed by the authors as a means of detecting audible Rub & Buzz which is an extreme type of distortion[1,2]. Both methods are based on the Perceptual Evaluation of Audio Quality (PEAQ) standard[3]. In the present work, in order to estimate the audibility of regular harmonic distortion, additional psychoacoustic variables are added to the CLEAR algorithm. These variables are then combined using an artificial neural network approach to derive a metric that is indicative of the overall audible harmonic distortion. Experimental results on headphones are presented to justify the accuracy of the model.

Authors: Steve Temme, Pascal Brunet and Parastoo Qarabaqi
Presented at the 133rd AES Convention, San Francisco, 2012

Full Paper

Headphone Testing (part 2)

/in /by
Graphic showing headphone testing part 2 article from Voice Coil Magazine

Headphone Testing Part 2

Author: Brian Fallon.  Reprinted from the April 2012 issue of Voice Coil.

In part 2 of this 2-part series, Brian Fallon discusses the challenges and practicalities of measuring headphones with built-in electronics such as digital and Bluetooth headphones. He covers connectivity, frequency limitations, test signals, measurement units and specialty measurements such as noise cancellation.

Full Article

Note: This article was written a few years back and there have been many advancements in the SoundCheck software and other accessories for measuring advanced headphones.

Read the first few paragraphs of Headphone Testing Part 2 Article

Introduction
“Headphone Testing (Part 1)” published in the December issue of Voice Coil, covered the basics of analog headphone testing: correction curves and fixturing, choosing hardware (ear simulators and couplers), the different requirements for testing in R&D vs. production, and the various essential measurements such as frequency response and distortion.

Analog headphones are relatively straightforward to test because there are only two electro-acoustic transducers to measure. Headphones with built-in electronics, such as digital headphones (including Bluetooth and USB), and noise-cancelling headphones are harder to test because the electronics and transducers need to be tested together as a complete system. In this article, test considerations for such headphone systems and the practicalities of testing them are discussed.

Testing Digital Headphones
Headphones with digital connectivity add complexity to audio testing. In addition to testing the acoustic transducers, the digital circuitry must also be considered. The fundamentals of analog headphone testing (the use of artificial ear simulators and couplers, the principals of repeatability vs. realism, and the tests that characterize the device) are the same, but the test signals must pass through the headphone electronics, which can greatly influence the test results. The principal distinction of digital headphones is that they contain a D/A converter and often DSP circuitry as well as a headphone amplifier. This means that unlike an analog headphone, where measurements are being conducted only on the electroacoustic elements, the measurements for digital headphones are being performed on the whole system which comprises everything from the digital signal to the acoustic output of the transducers. While it is certainly possible to isolate and test each of these components on its own, it is also very important to understand the intricacies of testing the complete device.

Managing Connectivity
The initial challenge of testing digital headphones is managing connectivity. The test system must be able to communicate directly with the device. A software-based system is ideal because it can communicate directly through the computer’s USB interface to the USB headphone, which will appear in the operating system along with other audio devices. Test signals can be sent digitally and the acoustic signals can be analyzed synchronously. If a hardware-based system is used, an extra program is usually required to connect the test system to the device under test.

Bluetooth headphones, however, require an additional interface for the computer to connect to the device under test. This may be a hardware Bluetooth communication box or a simple Bluetooth dongle, either built into the computer or externally connected by USB (see Figure 1). Bluetooth interfaces cause transmission delays in the audio chain. The test system must be able to account for these delays in order to take meaningful measurements. Some test systems can use an autodelay algorithm that looks at the system’s impulse response to calculate the delay and remove it from the measurement, if necessary.

Frequency Limitations
It is also important to be aware of frequency range limitations when designing tests for Bluetooth devices. Bluetooth devices typically operate at low-sampling rates of either 8 kHz (narrow band) or 16 kHz (wide band). These sampling rates limit the frequency range (because of the Nyquist frequency) to significantly narrower than analog headphones or even USB headphones. For example, a Bluetooth device with an 8-kHz sampling rate will only play audio up to slightly less than 4 kHz. Such limitations need to be considered when designing the test specifications. It can also be interesting to test the Bluetooth device beyond its cut-off frequency to see how well its anti-aliasing filter suppresses out-of-band signals.

Test Signals
Bluetooth presents a further challenge in that sine waves are not always transmitted accurately. When this occurs, alternative stimulus signals must be considered. Broadband noise is one possibility, but because of noise suppression circuits in some devices, this may not be a practical solution. A multitone signal, where several frequencies are played simultaneously, is another option. This produces a very fast frequency response measurement and is immune to the sudden dropouts that can occur in Bluetooth transmission. The downside of this test signal is that traditional harmonic distortion cannot be measured. Yet another possibility is the use of speech or music signals. These real-world audio signals transmit very well over Bluetooth, but their downsides are that they typically require long-term averaging and cannot be used for harmonic distortion measurements. If distortion measurement is required, non-coherent distortion may be measured using any test signal. This technique compares the input and output power spectrums to measure the non-coherent power and calculate the distortion plus noise (see Figure 2).

Measurement Units
Traditional analog headphones are tested with a stimulus level that is rated in terms of voltage or power. The sensitivity is also specified in these units such as dBSPL / mW. When testing headphones with USB or Bluetooth connectivity the stimulus is simply a digital signal whose level can be expressed in terms of digital full scale. The sensitivity is, therefore, expressed in dBSPL/FS. Manufacturers sometimes choose to relate these signal levels back to voltage, which can be done if the characteristics of the D/A circuit are known. In such cases, the gain of the built-in headphone amplifier chip must also be accounted for. Another method used for relating these digital units back to the analog domain is through the use of a codec (see Figure 3). A-law and M-law are two codecs widely-used in Bluetooth applications that can translate the digital units into “virtual volts.” These codecs are most commonly used in telecommunications, especially for headsets.

Read the Full Article

Check out our main Headphone Testing page for up-to-date information on testing advanced headphones.