Tag Archive for: smart speakers

Smart Device Testing Seminar

Testing smart devices such as smart speakers, hearables, watches and more involves a combination of tests from many different application areas, ranging from simple transducer tests to open loop tests, Bluetooth measurements, voice activation tests, telecoms standards, hearing aid measurements and more. In this demo-focused seminar, we demonstrate how SoundCheck makes all the measurements you need throughout the entire smart device design process from component selection to pre-qualification testing.

Smart device testing demonstrations include:

  • Component tests: MEMS microphone measurement, directional microphone measurement, speaker evaluation
  • Prototype tests – open loop speaker test on a smart speaker, open loop microphone test on a smart speaker, Bluetooth hearables test, headphone Active Noise Cancellation (ANC) Measurement
  • Background Noise Tests: calibration, using pub noise to evaluate a microphone’s ability to reject background noise

Presenters: Steve Temme, Mark Latshaw, Steve Tatarunis
Duration: 48 Mins

Smart Device Measurement Resources

This seminar was originally broadcast on June 6th 2023. The recording below does not include the live Q&A at the end for attendee confidentiality reasons. However, several links to additional resources were provided during the Q&A session, and these are provided below.

  1. Internally routing audio: Open Loop microphone testing requires audio to be routed into SoundCheck. We recommend:
    1. Virtual Audio Cable for Windows: https://vac.muzychenko.net/en/index.htm
    2. Soundflower for Mac: https://github.com/mattingalls/Soundflower
  2. In response to a question about whether haptic feedback could be measured in a smartwatch, we shared a SoundCheck test sequence for measuring haptic feedback: https://www.listeninc.com/products/test-sequences/free/linear-motor-test-sequence/
  3. We touched briefly on communications audio tests when smart speakers are used as hands-free devices. To learn more about Communications Testing techniques and standards, please see our Communications Testing Seminar.
  4. Similarly, in answer to a question about testing automotive infotainment systems, we shared our Automotive audio testing seminar

More about how to Measure Smart Devices

Check out our main page on Smart Device Testing, which includes links to test sequences, relevant products and more.

 

Practical Testing of Voice Controlled Devices

Author: Steve Temme.  Reprinted from the Jan 2020 issue of AudioXpress.

This article discusses tools and techniques that are available to accurately measure the audio performance of voice-controlled and connected devices under the many various real-world conditions they may be used. It covers basic acoustic measurements such as frequency and distortion response, which have always been carried out on conventional wired systems, and the more complex real-world tests that apply specifically to voice-activated devices, along withthe techniques and standards that may be used.
Full Article

 

 

 

 

 

The Challenges of Testing Voice-Controlled Audio Systems

Testing voice-controlled audio systems such as smart speakers, hearables, and vehicle infotainment systems is notoriously complex. They have numerous connections from wired to wireless and contain much signal processing, both on the record and the playback side. This means that their characteristics change according to ‘real world’ conditions of the environment that they are used in, such as background noise, playback levels, and room acoustics. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback, operation as a hands-free telephone, and in the case of hearables, hearing assistance. Due to their complex non-linear use cases, these devices often need to be tested at different levels and different environmental conditions. This paper focuses on tools and techniques to accurately measure the audio performance of such devices under the many various real-world conditions in which they are used.

 

语音控制的智能设备(例如智能扬声器、听觉设备和车辆信息娱乐系统)非常难以测试。它们具有从有线到无线的多样连接方式,并且在接收端和重放端使用了诸多信号处理技术。这意味着它们的特性会随着使用环境的“现实世界”条件(例如背景噪声、播放级别和室内声学条件)的不同而变化。 此外,它们的多功能特性意味着可能需要测试该设备的许多方面,包括语音识别、音乐播放、作为免提电话或听觉设备或助听器使用时的性能。由于其复杂的非线性使用情况,这些设备通常需要在不同级别和不同环境条件下进行测试。本文重点介绍在各种实际条件下准确测量此类设备的音频性能的工具和技术。

Author: Steve Temme, Listen, Inc.
Presented at ISEAT 2019, Shenzhen, China.

Full Paper – English Version
Full Paper – Chinese Version

 

Paper Preview: The Challenges of Testing Voice-Controlled Audio Systems

Abstract
Smart devices that are voice-controlled such as smart speakers, hearables, and vehicle infotainment systems are notoriously complex to test. They have numerous connections from wired to wireless and contain much signal processing, both on the record and the playback side. This means that their characteristics change according to ‘real world’ conditions of the environment that they are used in, such as background noise, playback levels, and room acoustics. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback, operation as a hands-free telephone, and in the case of hearables, hearing assistance. Due to their complex non-linear use cases, these devices often need to be tested at different levels and different environmental conditions. This paper focuses on tools and techniques to accurately measure the audio performance of such devices under the many various real-world conditions in which they are used.

Keywords: hearables, automotive infotainment, smart speakers, smartphones, test

Introduction
Smart Devices such as smart speakers, hearables and automotive infotainment systems have become increasingly challenging to test. They have many possible interfaces ranging from hardwired to wireless (Bluetooth, cloud-based), smartphone, voice (“Hey Siri”, “OK Google”, “Alexa”), and in the case of automotive, even USB memory stick and CarPlay/Android Auto. There is usually much signal processing, both on the record side (e.g. beamforming, background noise filtering, voice activity detection, and on the playback side (e.g. loudness, compression, equalization, and active noise cancellation).
This means that their characteristics change according to ‘real world’ conditions such as the physical environment and background noise. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback or even operation as a hands-free telephone, telephone headset or hearing aid. These devices often need to be tested at different levels and in different environmental conditions, for example different physical setups and with/without background noise, different signals etc.

Although, there are currently no standards for testing most smart devices, principles and test configurations are borrowed from many other audio devices and use existing standards such as IEC and BS EN for loudspeakers and headphones , IEEE for headsets, IEEE/TIA/ITU for telephone test, ANSI and IEC for hearing aid standards , and ETSI for background noise. Flexibility of the test system and experience with testing a wide range of acoustic devices is therefore critical to enable a device to be completely characterized.

This paper explains how to implement both basic acoustic tests and more complex real-world tests along with the techniques and standards that may be used. Most of the tests discussed are relevant to all smart devices including smart speakers, hearables and automotive infotainment, but some hearable-specific additional tests are also detailed. Finally, we present a check list of the test-system functionality you should look for when choosing a system to fully characterize a smart speaker or other smart device.

 

Full Paper – English Version
Full Paper – Chinese Version

 

More about testing infotainment systems.

Triggered Record Using Chirp Trigger and WAV File (Version 17 and later)

This test sequence demonstrates SoundCheck’s Triggered Record – Chirp Trigger function for open loop testing of devices without analog inputs such as smart speakers, wearables, smart home devices, tablets and cellphones.  A stimulus WAV file is created in SoundCheck and transferred to the device under test, where it is played back and the response recorded in SoundCheck as if the stimulus were played directly from SoundCheck. The Acquisition step is triggered by the chirp in the stimulus file. Chirp triggers are more robust than level and frequency triggers which are susceptible to false triggering due to background noise.

More

Listen offers ETSI Standard background noise generation system

Listen offers ETSI Standard background noise generation system

Listen, Inc. has launched the ETSI standard background noise generation module, a SoundCheck test sequence which calibrates a 4.1 speaker array to conform with the ETSI ES 202 396-1 Standard. This provides an equalized, calibrated playback solution to stress devices in a standardized and repeatable way. The sequence includes a library of real-world binaural recordings from the ETSI standard: cafeteria, pub, crossroad, vehicle, single voice distractor, and office noises.  Custom or user-defined binaural recordings can also be used. With a purchase price of just $5,000 for the test sequence, this offers an extremely economical alternative to a conventional $20,000+ stand-alone background noise generation system.

Furthermore, it offers the advantage that the ETSI standard library is fully integrated with, and managed by, the SoundCheck test system. This means that the calibrated background noise levels can be fully controlled and adjusted as part of a pre-programmed test sequence, significantly reducing test development time as well as simplifying the physical setup.

This has many applications, for example evaluating ANC, noise suppression, SNR optimization of microphone arrays, beamforming directionality studies, and more. It is particularly useful for voice recognition testing as the test sequence may include loops to incrementally increase the volume or change the noise and repeat the test until the voice is no longer accurately recognized.

More Information

Triggered Record Using WAV File (Version 16.1 and later)

This sequence allows you to test devices without an analog input such as smart speakers, tablets, cellphones and MP3 players using SoundCheck’s frequency-based trigger functionality. This method offers improved accuracy over previous level-based triggering, especially in noisy environments. A stimulus WAV file is created in SoundCheck, and copied to the device under test, where it is played and the response recorded in SoundCheck as if the stimulus were played directly from SoundCheck. The stimulus WAV file to be used on the device under test (DUT) may be customized in the stimulus step.

Note that this sequence uses the level-based trigger available in SoundCheck 16.1 and later. If you are using version 16.0 or earlier, please see the level-based trigger sequence.

More

 

Smarter Measurements for Smart Speakers

Author: Daniel Knighten (Listen, Inc) and Glenn Hess (Indy Acoustic Research).  Reprinted from the March 2018 issue of Audio Xpress.

In this article, we describe techniques to characterize the frequency response, output level, and distortion of the device under test to enable direct comparisons between Internet of Things (IoT) smart speakers and conventional speakers.

Full Article

Smart Speaker – Embedded Microphone Test Sequence

smart_speaker_final_display_micThis sequence demonstrates a method by which SoundCheck can measure the performance of a microphone embedded in a so-called “smart speaker”. This example assumes that the DUT is an Amazon Echo but it can be adapted for use with virtually any other type of smart speaker by substituting the Echo’s voice activation phrase WAV file (“Alexa”) with one specific to the desired make and model.

The sequence begins by playing a voice activation phrase out of a source speaker, prompting the DUT to record both the voice command and the ensuing stepped sine sweep stimulus. A message step then prompts the operator to retrieve this recording from the DUT’s cloud storage system. This is accomplished by playing back the recording from the cloud and capturing it with a Triggered Record step in the SoundCheck test sequence.  The Recorded Time Waveform is then windowed (to remove the voice command) and frequency shifted prior to analysis and the result (Frequency Response) is shown on the final display step.

More

Smart Speaker – Embedded Loudspeaker Test Sequence

smart_speaker_final_displayThis sequence demonstrates a method by which SoundCheck can measure the performance of a loudspeaker embedded in a so-called “smart speaker”. This example assumes that the DUT is an Amazon Echo but it can be adapted for use with virtually any other type of smart speaker by substituting the Echo’s voice activation phrase audio file (“Alexa, play Test Signal One”) with one specific to the desired make and model.

The sequence begins by playing the voice activation phrase out of a source speaker, prompting the DUT to playback the mp3 stimulus file from the cloud, followed by a pause step to account for any activation latency. Following the pause, a triggered record step is used to capture the playback from the DUT. The Recorded Time Waveform is then frequency shifted prior to analysis and the results (Frequency Response, THD and Perceptual Rub & Buzz) are shown on the final display step.

We recommend reading our AES paper on this subject prior to continuing as it contains additional details on the test methods devised for this sequence.

More

Challenges of IoT Smart Speaker Testing

Quantitatively measuring the audio characteristics of IoT (Internet of Things) smart speakers presents several novel challenges. We discuss overcoming the practical challenges of testing such devices and demonstrate how to measure frequency response, distortion, and other common audio characteristics. In order to make these measurements, several measurement techniques and algorithms are presented that allow us to move past the practical difficulties presented by this class of emerging audio devices. We discuss test equipment requirements, selection of test signals and especially overcoming the challenges around injecting and extracting test signals from the device.

Authors: Glenn Hess (Indy Acoustic Research) and Daniel Knighten (Listen, Inc.)
Presented at the 143rd AES Conference, New York 2017

Full Paper