The Challenges of Testing Voice-Controlled Audio Systems

Smart devices that are voice-controlled such as smart speakers, hearables, and vehicle infotainment systems are notoriously complex to test. They have numerous connections from wired to wireless and contain much signal processing, both on the record and the playback side. This means that their characteristics change according to ‘real world’ conditions of the environment that they are used in, such as background noise, playback levels, and room acoustics. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback, operation as a hands-free telephone, and in the case of hearables, hearing assistance. Due to their complex non-linear use cases, these devices often need to be tested at different levels and different environmental conditions. This paper focuses on tools and techniques to accurately measure the audio performance of such devices under the many various real-world conditions in which they are used.


语音控制的智能设备(例如智能扬声器、听觉设备和车辆信息娱乐系统)非常难以测试。它们具有从有线到无线的多样连接方式,并且在接收端和重放端使用了诸多信号处理技术。这意味着它们的特性会随着使用环境的“现实世界”条件(例如背景噪声、播放级别和室内声学条件)的不同而变化。 此外,它们的多功能特性意味着可能需要测试该设备的许多方面,包括语音识别、音乐播放、作为免提电话或听觉设备或助听器使用时的性能。由于其复杂的非线性使用情况,这些设备通常需要在不同级别和不同环境条件下进行测试。本文重点介绍在各种实际条件下准确测量此类设备的音频性能的工具和技术。

Author: Steve Temme, Listen, Inc.
Presented at ISEAT 2019, Shenzhen, China.

Full Paper – English Version
Full Paper – Chinese Version

Listen Releases SoundCheck 17: New Features for Multichannel and Voice-Controlled Testing

Save to MATLAB

Listen is excited to announce the release of SoundCheck Version 17. This new Windows/Mac release offers many features to simplify multi-channel and voice controlled testing, such as a new level and cross-correlation trigger, average curve/waveform post-processing functionality, the ability to easily read and work with multichannel wavefiles, multiple DC Connect control, and enhanced database options. On the usability side, SoundCheck 17 offers increased flexibility in color palettes, save to MATLAB option, and the ability to recall CSV formatted text files.

Today’s modern audio devices have two important testing requirements: the ability to test a voice activated device with no analog input, and the ability to control and test multiple channels simultaneously, for example, microphone or speaker arrays. SoundCheck 17 contains a host of new features to facilitate this. For voice-activated measurements on devices with no analog input, such as smart speakers, wearables, hearables and smart home devices, the new level & cross-correlation ‘smart trigger’ offers improved performance. By using a chirp-based conditioning tone and searching for the exact log sweep frequency, it is more robust and less susceptible to false triggers than simpler level and frequency triggers.

To save time when testing smart devices where it is necessary to test microphone or speaker arrays, SoundCheck 17 can now directly read multichannel WAV files from the memory list, signal generator and stimulus steps. Also for testing multi-channel arrays, multiple DC Connects can now be controlled, independently configured, and used for data acquisition within SoundCheck. The new Average Curve/WFM post processing function which allows the average curve (or waveform) of a selected group of data in the memory list to be obtained, is particularly useful for power averaging selected curves, averaging curves from different spatial positions (e.g. microphone arrays), and complex averaging of multiple measurements with background noise. Finally, the database module has been enhanced and is now it 4x faster and 3x more space-efficient than previous versions. This is particularly important for testing modern audio products with multiple transducers (microphone arrays, multiple speakers, etc.), as these often generate large volumes of data.

New Level and Cross-Correlation Trigger

Usability enhancements include improved color pallets which offer ultimate flexibility in defining colors for backgrounds, grids, cursors and graph lines. New default color palettes are included, and user-defined palettes can be saved as pre-set files which will be applied to any new display created. Multiple palettes can be saved, for example allowing different color sets for different applications. In addition, SoundCheck can now save any data (including memory list curves and Soundmap (time frequency analysis) data ) to  MATLAB for additional processing. Data is saved as a standard MAT file and can be manually or automatically saved in this format.

Additional new features include support for the new APTX HD codec for high resolution Bluetooth testing, a 64 bit Demo / Data Viewer which enables measured data to be recalled and viewed without the need to own a separate license, and the ability to recall CSV files as well as TXT files.


More Information:

Listen offers ETSI Standard background noise generation system

Listen offers ETSI Standard background noise generation system

Listen, Inc. has launched the ETSI standard background noise generation module, a SoundCheck test sequence which calibrates a 4.1 speaker array to conform with the ETSI ES 202 396-1 Standard. This provides an equalized, calibrated playback solution to stress devices in a standardized and repeatable way. The sequence includes a library of real-world binaural recordings from the ETSI standard: cafeteria, pub, crossroad, vehicle, single voice distractor, and office noises.  Custom or user-defined binaural recordings can also be used. With a purchase price of just $5,000 for the test sequence, this offers an extremely economical alternative to a conventional $20,000+ stand-alone background noise generation system.

Furthermore, it offers the advantage that the ETSI standard library is fully integrated with, and managed by, the SoundCheck test system. This means that the calibrated background noise levels can be fully controlled and adjusted as part of a pre-programmed test sequence, significantly reducing test development time as well as simplifying the physical setup.

This has many applications, for example evaluating ANC, noise suppression, SNR optimization of microphone arrays, beamforming directionality studies, and more. It is particularly useful for voice recognition testing as the test sequence may include loops to incrementally increase the volume or change the noise and repeat the test until the voice is no longer accurately recognized.

More Information

Seminars in China: Smart Speakers and Automotive Audio

The Challenges of Testing Speech Controlled Audio Systems

Your devices got smarter. Did your test system?

Learn how to implement open loop tests for both playback and recording in a range of devices including smart speakers, automotive audio, robots, IoT devices and more in this practical seminar.

Open loop testing (testing devices where inputs and outputs are independent) enables many types of smart devices and their components to be tested in various formats and situations including:

  • Smart speakers, smart watches and other smart devices
  • Microphone arrays
  • Speech recognition systems with microphones
  • In-vehicle audio systems
  • Audio devices/systems with no physical inputs or outputs
  • Testing in noisy environments

We explain how to measure the same parameters as traditional loudspeakers, discussing such challenges as injecting and extracting response signals, time delays, and triggering the system using voice commands. We also demonstrate how to measure microphone array performance, speakerphone performance and more. Course content includes:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling

Speakers /主讲人: Steve Temme, Listen, Inc. and Peter Wulf Andersen, GRAS.

Date & Locations /时间与地点:

2019年1月15日(Tuesday) – Taipei台北
2019年1月16日(Wednesday) – Shenzhen深圳
2019年1月18日(Friday) – Suzhou苏州

More information and registration