Practical Testing of Voice Controlled Devices
This article discusses tools and techniques that are available to accurately measure the audio performance of voice controlled devices under the many various real-world conditions they may be used. It covers basic acoustic measurements such as frequency and distortion response, which have always been carried out on conventional wired systems, and the more complex real-world tests that apply specifically to voice-activated devices, along with the techniques and standards that may be used.
Author: Steve Temme. Reprinted from the Jan 2020 issue of AudioXpress.
Voice Controlled Devices – Audio Testing and Key Considerations
This briefing document reviews key themes and important takeaways from the provided excerpts of “Practical Testing of Voice-Controlled Audio Devices” (ax, January 2020). The document focuses on the complexities of testing voice-controlled audio devices, including smart speakers, hearables, and automotive infotainment systems, and highlights relevant industry standards and test system recommendations.
Main Themes:
- Complexity of Testing: Unlike traditional audio devices, voice-controlled devices present unique testing challenges due to:
- Diverse interfaces (Bluetooth, cloud, voice assistants)
- Extensive signal processing (beamforming, noise cancellation)
- Multifunctional nature (voice recognition, music playback, communication)
- Real-world condition dependence (background noise, playback levels)
- Importance of Real-World Testing: The article emphasizes the need to test devices under various real-world conditions. Factors like background noise, playback levels, and user interaction significantly impact performance and should be considered during testing.
- Leveraging Existing Standards: While specific standards for smart device testing are limited, the article recommends adopting guidelines from similar devices like speakers, headphones, hearing aids, and telephones (IEC, EN, IEEE, TIA, ITU, ANSI, ETSI).
- Open-Loop Measurement Challenges: Open-loop systems, characteristic of smart devices, lack a synchronous signal path, leading to sampling rate errors. The article explains how algorithms utilizing triggers and heterodyne filtering correct these errors for accurate analysis.
- Cloud-Based Measurement Complexities: Testing involves interacting with devices like a real user, utilizing cloud services for playback and recording. The article describes the setup and intricacies of testing via the cloud, including voice commands, signal routing, and manufacturer-specific ecosystems.
Important Ideas/Facts:
- Basic Acoustic Tests: Essential for measuring frequency response, distortion, and stereo tracking in both wired and wireless modes. Requires calibrated equipment like mouth simulators, measurement microphones, and ear simulators (HATS recommended for hearables and infotainment systems).
- Open-Loop Measurements: A technique to correct sampling rate errors inherent in open-loop systems. The article details the process of using trigger signals, heterodyne filtering, and resampling for accurate frequency and distortion analysis.
- Measurements via the Cloud: This involves using voice commands, proprietary voice services, and cloud platforms for signal injection and extraction. The article explains the setup and challenges of testing various smart device functionalities through the cloud.
- Nonlinear Behavior: The performance of smart devices varies based on factors like background noise type and volume, music playback, and voice communication. The article highlights the need to account for this nonlinearity during testing.
- Speech Recognition Testing: Evaluating the accuracy of voice command understanding is crucial. The article recommends using “Harvard Sentences,” a phonetically balanced sentence set, as track titles for testing different voices, accents, and background noise conditions.
- Background Noise Simulation: Crucial for testing voice recognition, SNR optimization, and beamforming. The article recommends using ETSI standard ES 202 396-1, which provides binaural recordings of various noise environments, to create realistic test scenarios.
- Voice Quality Evaluation: For devices functioning as hands-free communication tools, the article suggests using telephone test sequences like TIA standard TIA920, which covers both hands-free and headset devices.
- Noise Cancellation Assessment: Especially relevant for hearables and infotainment systems. The article outlines the procedure for measuring passive isolation, active attenuation, and total attenuation using a calibrated background noise system.
- Hearing Assistance Measurement: For hearables with hearing enhancement features, the article recommends testing amplification and compression algorithms according to ANSI/IEC hearing aid standards using anechoic test boxes and artificial ears.
Test System Recommendations:
The article concludes with a checklist of functionalities to consider when selecting a test system for voice-controlled audio devices:
- Input and output component testing
- Open-loop testing capabilities
- Speech and music signal support
- Background noise generation
- Cloud signal routing (virtual audio cable)
- Pre-written test sequences (TIA)
- Flexible test programming
- Calibrated Bluetooth interface
- Automotive bus compatibility (e.g., A2B)
By understanding the complexities and utilizing appropriate testing methodologies, manufacturers can ensure optimal audio performance and user experience in voice-controlled audio devices.
In this article, we describe techniques to characterize the frequency response, output level, and distortion of the device under test to enable direct comparisons between Internet of Things (IoT) smart speakers and conventional speakers.
