Posts

The Challenges of Testing Voice-Controlled Audio Systems

Smart devices that are voice-controlled such as smart speakers, hearables, and vehicle infotainment systems are notoriously complex to test. They have numerous connections from wired to wireless and contain much signal processing, both on the record and the playback side. This means that their characteristics change according to ‘real world’ conditions of the environment that they are used in, such as background noise, playback levels, and room acoustics. Furthermore, their multifunctional nature means that there are many aspects of the device that may need to be tested, ranging from voice recognition to music playback, operation as a hands-free telephone, and in the case of hearables, hearing assistance. Due to their complex non-linear use cases, these devices often need to be tested at different levels and different environmental conditions. This paper focuses on tools and techniques to accurately measure the audio performance of such devices under the many various real-world conditions in which they are used.

 

语音控制的智能设备(例如智能扬声器、听觉设备和车辆信息娱乐系统)非常难以测试。它们具有从有线到无线的多样连接方式,并且在接收端和重放端使用了诸多信号处理技术。这意味着它们的特性会随着使用环境的“现实世界”条件(例如背景噪声、播放级别和室内声学条件)的不同而变化。 此外,它们的多功能特性意味着可能需要测试该设备的许多方面,包括语音识别、音乐播放、作为免提电话或听觉设备或助听器使用时的性能。由于其复杂的非线性使用情况,这些设备通常需要在不同级别和不同环境条件下进行测试。本文重点介绍在各种实际条件下准确测量此类设备的音频性能的工具和技术。

Author: Steve Temme, Listen, Inc.
Presented at ISEAT 2019, Shenzhen, China.

Full Paper – English Version
Full Paper – Chinese Version

Triggered Record Using Chirp Trigger and WAV File (Version 17 and later)

This test sequence demonstrates SoundCheck’s Triggered Record – Chirp Trigger function for open loop testing of devices without analog inputs such as smart speakers, wearables, smart home devices, tablets and cellphones.  A stimulus WAV file is created in SoundCheck and transferred to the device under test, where it is played back and the response recorded in SoundCheck as if the stimulus were played directly from SoundCheck. The Acquisition step is triggered by the chirp in the stimulus file. Chirp triggers are more robust than level and frequency triggers which are susceptible to false triggering due to background noise.

More

Listen offers ETSI Standard background noise generation system

Listen offers ETSI Standard background noise generation system

Listen, Inc. has launched the ETSI standard background noise generation module, a SoundCheck test sequence which calibrates a 4.1 speaker array to conform with the ETSI ES 202 396-1 Standard. This provides an equalized, calibrated playback solution to stress devices in a standardized and repeatable way. The sequence includes a library of real-world binaural recordings from the ETSI standard: cafeteria, pub, crossroad, vehicle, single voice distractor, and office noises.  Custom or user-defined binaural recordings can also be used. With a purchase price of just $5,000 for the test sequence, this offers an extremely economical alternative to a conventional $20,000+ stand-alone background noise generation system.

Furthermore, it offers the advantage that the ETSI standard library is fully integrated with, and managed by, the SoundCheck test system. This means that the calibrated background noise levels can be fully controlled and adjusted as part of a pre-programmed test sequence, significantly reducing test development time as well as simplifying the physical setup.

This has many applications, for example evaluating ANC, noise suppression, SNR optimization of microphone arrays, beamforming directionality studies, and more. It is particularly useful for voice recognition testing as the test sequence may include loops to incrementally increase the volume or change the noise and repeat the test until the voice is no longer accurately recognized.

More Information

Seminars in China: Smart Speakers and Automotive Audio

The Challenges of Testing Speech Controlled Audio Systems

Your devices got smarter. Did your test system?

Learn how to implement open loop tests for both playback and recording in a range of devices including smart speakers, automotive audio, robots, IoT devices and more in this practical seminar.

Open loop testing (testing devices where inputs and outputs are independent) enables many types of smart devices and their components to be tested in various formats and situations including:

  • Smart speakers, smart watches and other smart devices
  • Microphone arrays
  • Speech recognition systems with microphones
  • In-vehicle audio systems
  • Audio devices/systems with no physical inputs or outputs
  • Testing in noisy environments

We explain how to measure the same parameters as traditional loudspeakers, discussing such challenges as injecting and extracting response signals, time delays, and triggering the system using voice commands. We also demonstrate how to measure microphone array performance, speakerphone performance and more. Course content includes:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling

Speakers /主讲人: Steve Temme, Listen, Inc. and Peter Wulf Andersen, GRAS.

Date & Locations /时间与地点:

2019年1月15日(Tuesday) – Taipei台北
2019年1月16日(Wednesday) – Shenzhen深圳
2019年1月18日(Friday) – Suzhou苏州

More information and registration

West Coast Training Extravaganza! Feb 4-8, 2019

Listen is excited to announce its biggest ever training event on the West Coast the week of Feb 4th– 8th (Chinese New Year week) with 5 full days of training events.

Monday Feb 4th: Electroacoustics training with Chris Struck. This one-day course covers the fundamentals of electroacoustics and is ideal for anyone new to the industry or who wants to learn more about the electroacoustics theory behind the measurements they are making. You do not need to be a SoundCheck user to benefit from this class! Course Outline. Register.

Tuesday and Wednesday Feb 5th & 6th: Basic 2 day SoundCheck training with Steve Tatarunis. This two-day class is designed to accelerate the learning curve for novice and basic level SoundCheck users. A combination of teaching, demos and hands-on exercises begin with the basics of system setup and calibration, using virtual instruments, and creating and modifying basic test sequences. On the second day, the sequence writing progresses to more complex techniques such as sequence loops, statistics and advanced post-processing. Course Outline. Register

Thursday Feb 7th: Advanced SoundCheck training with Steve Temme. This is designed for users who have already attended the basic class or who need to make more complex measurements and want to take their SoundCheck use to the next level. This course will include topics such as simulated free field measurements, open loop testing of smart devices, advanced distortion measurements, selecting an appropriate test stimulus (including speech, music and compound stimuli), advanced sequence development including conditional branching and looping, advanced post-processing algorithms and more. More details to follow. Register

Friday Feb 8th:  Smart speaker testing seminar (half day plus networking lunch). By popular demand we are bringing our free smart speaker testing seminar back to the West Coast. Steve Temme will present this seminar covering many aspects of smart device testing including open loop measurement, telephony measurements and voice quality measurements. Although the demonstrations will be done with smart speakers, the techniques and methods are relevant to anyone testing voice-controlled devices including hearables, watches and other wearables, robots, smart home devices and automotive entertainment systems. FREE. Course Outline. Register.

Scroll down for full class outlines.

Location: Hilton Garden Inn, 10741 N Wolfe Rd, Cupertino, CA 95014.

Costs:
We have simplified pricing this year so that each day is the same cost and you can combine any number of days, with a discount for each additional day that you take. In addition, support contract holders and multiple attendees from the same company receive additional discounts

1 Day: $495 ($350 with valid SoundCheck support contract)

2 Days: $950 ($665 with valid SoundCheck support contract)

3 Days: $1350 ($950 with valid SoundCheck support contract)

4 Days: $1725 ($1200 with valid SoundCheck support contract)

Smart Speaker Seminar: FREE

 

Multi-person discount:

2-3 people from one company: 10% discount;

4 people will receive a 15% discount

5+ will receive a 25% discount.

Note: to qualify for the discount, all attendees must register together. Multi-attendee discount is only applicable to Mon-Thurs training since the Friday Smart Speaker Seminar event is free.

 

1- Day Electroacoustics Class Outline. Instructor – Christopher Struck (CJS Labs)

Introduction

    • Transducers & Reciprocity
    • Subjective & Objective Tests
    • Loudspeaker and Microphone Test Systems

Basic Acoustics and Measuring Sound

  • Sound Pressure & Sound Pressure Level
  • Wavelength & Frequency
  • The Decibel and Working with Decibels
  • Weighting Networks
  • Detectors, and Time Constants

Sound Sources and Sound Fields

  • Ideal Sources, Sound Radiation, and Sound Fields
  • Transmission, Reflection and Diffraction
  • Elementary Room Acoustics
  • Anechoic Chambers and Reverb Rooms

Psychoacoustics & Hearing

  • Frequency and Dynamic Range
  • Masking & Critical Bands
  • Directional Effects
  • Anatomy of the Ear
  • Detection and Signal Processing within the Ear
  • Loudness & Hearing Loss

Measurement Microphones

  • Principles of Operation
  • Frequency & Dynamic Range
  • Microphone Types, Selection and Applications
  • Field Calibration

Frequency Analysis

  • Signal Types
  • Filters and Filter Analysis
  • Confidence Limits, and Averaging Time
  • The FFT
  • Time Windows, Overlap & “Real Time” Analysis

 System Analysis

  • Time & Frequency Domains
  • Causal Linear Time-Invariant Systems
  • Analysis Methods: Dual Channel FFT, MLS, TDS
  • Stimulus Signals

Simulated Free Field Measurements

  • Sound Paths, Correlated and Uncorrelated Noise
  • Quasi-Anechoic Test Methods
  • Near Field Measurements
  • Time Selective Techniques

Distortion & Non-Linearity

  • Non-Linear Systems & Distortion Order
  • Distortion Audibility
  • Transducer Defect Detection
  • Harmonic, Intermodulation & Difference Frequency Distortion Tests
  • Measurements of Band-Limited Systems
  • Multi-Tone Distortion
  • THD + N
  • Signal-to-Noise Ratio and Non-Coherent Power
  • Time Variance and Linearity Measures

2-Day Basic SoundCheck Training: Class Outline. Instructor: Steve Tatarunis (Listen, Inc.)

In this class, we start with the basics – how to set up your system, calibrate your hardware and use the virtual instruments.  We then focus on how to develop and modify test sequences, starting with stimulus, analysis and display options, and progressing to more advanced functionality such as sequence loops and jumps, statistics, and advanced post-processing. The class is a combination of classroom-style teaching and hands on practical sessions, so that when you leave the class you will be ready to go in practice as well as in theory.

This hands-on class covers all the major features of SoundCheck, including writing sequences, test signals, analysis, distortion measurements and more. The class is a combination of presentations interspersed with hands-on exercises to ensure that you are comfortable with all the concepts and methods taught before you leave the classroom. Although no previous SoundCheck experience is assumed, the course still offers plenty to users familiar with writing basic sequences.  Content includes:

  • Overview of SoundCheck software
    • Software Options
  • Hardware
    • Configuring audio interfaces and external equipment
    • Listen Certified Sound Cards
  • Calibration
    • Calibration of different DUT types (telephones, speakers, etc.)
  • Virtual instruments
    • Spectrum Analyzer, RTA, Multi-meter, Signal Generator, Oscilloscope
    • Virtual Instruments – hands-on
  • Online forum, bug and wish lists
  • SoundCheck Steps & Sequences
    • Philosophy & Architecture
    • Creating and editing SoundCheck Test Sequences and Steps
    • Using the Sequence Wizard
    • Stimulus & Acquisition Step details
    • Basic analysis
    • Display
    • Data Management Hands-on
    • Limits
    • Sequence Development – hands-on
    • Printing
    • Serial numbers
    • Autosave and recall
    • Messages – user prompts, calling external devices
    • Post-processing
    • Sequence Development (part 2) – hands-on
    • Statistics
    • Sequence loops and jumps
    • Advanced sequence development

Note: This is intended as a 2-day class, but each day may be taken separately if wanted. The first day will focus more on setup, calibration, virtual instruments and some beginning sequence writing. The second day will focus on sequence writing in much more detail.

 

1-Day Advanced SoundCheck Training: Class Outline. Instructor Steve Temme

This is designed for users who have already attended the basic class or who need to make more complex measurements and want to take their SoundCheck use to the next level. This course will include topics such as simulated free field measurements, open loop testing of smart devices, advanced distortion measurements, selecting an appropriate test stimulus (including speech, music and compound stimuli), advanced sequence development including conditional branching and looping, advanced post-processing algorithms and more. WE are finalizing content and a more detailed outline will follow shortly.

 

 

Free Seminar: Smarter Measurements for Smarter Speakers! Instructor Steve Temme (Listen, Inc.)

smart_speakersLearn how to test smart speakers, hearables, watches and other wearables, robots, smart home devices, automotive entertainment systems and other smart devices in our practical half-day seminar taught by industry expert Steve Temme of Listen, Inc.

Although acoustic testing of smart speakers and other voice-activated devices presents challenges such as injecting and extracting response signals, time delays, and triggering the system using voice commands, it is still possible to measure the same parameters as traditional loudspeakers, In addition we can also measure microphone array performance, speakerphone performance and more.

We demonstrate how to do this, focusing on:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling

There is no charge for this seminar and lunch will be provided.

Print

Triggered Record Using WAV File (Version 16.1 and later)

This sequence allows you to test devices without an analog input such as smart speakers, tablets, cellphones and MP3 players using SoundCheck’s frequency-based trigger functionality. This method offers improved accuracy over previous level-based triggering, especially in noisy environments. A stimulus WAV file is created in SoundCheck, and copied to the device under test, where it is played and the response recorded in SoundCheck as if the stimulus were played directly from SoundCheck. The stimulus WAV file to be used on the device under test (DUT) may be customized in the stimulus step.

Note that this sequence uses the level-based trigger available in SoundCheck 16.1 and later. If you are using version 16.0 or earlier, please see the level-based trigger sequence.

More

 

Seminar: Smarter Measurements for Smarter Speakers – Denmark

Smarter speakers require smarter test methods!

smart_speakersLearn how to test smart speakers, robots, voice-controlled automotive audio and other IoT devices in our practical half-day seminar taught by industry expert Steve Temme of Listen, Inc., with additional information on microphone and hardware products for testing these devices from G.R.A.S.

Although acoustic testing of smart speakers and other voice-activated devices presents challenges such as injecting and extracting response signals, time delays, and triggering the system using voice commands, it is still possible to measure the same parameters as traditional loudspeakers, In addition we can also measure microphone array performance, speakerphone performance and more.

We demonstrate how to do this, focusing on:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling
  • Microphones, artificial mouths and other hardware for smart speaker testing

There is no charge for this one-day seminar and lunch will be provided. There will also be a factory tour of G.R.A.S. Space is limited, so please RSVP today.

Print

Location and Date:

Holte (Copenhagen area), Denmark. Friday September 28th, 2018 – 9.00am – 3.00pm. GRAS Sound & Vibration A/S, Skovlytoften 33, DK-2840 Holte, Denmark

Agenda (9am-3.00pm)

  • Introduction
  • Equipment for Smart Speaker Testing
    • Listen Product Overview
    • GRAS Product Overview
      • Choosing the right microphone for smart speaker testing environments (far field vs near field vs unconventional environments)
      • When to use a Mouth Simulator vs Head and Torso Simulator for testing
  • Open Loop Testing
    • What is ‘open loop’ testing?
    • Can you really measure frequency response and harmonic distortion in an open loop?
    • Using Frequency Shifting to time align asynchronous stimuli and response
    • Working with cloud based services
  • Testing with Real World Signals
    • How to use real voice and / or non test tones to evaluate acoustic performance
    • Analysis techniques used for evaluating non test tone stimuli
  • Voice Recognition
    • Key Word spotting / Wake Up word testing
    • Room effects, environmental distractors, and SNR…how they affect voice recognition performance
    • Measuring directionality
  • Networking lunch
  • Telephony
    • What voice quality metrics are important for speakerphones?
    • Challenges of testing speakerphones in different environments
  • GRAS Factory tour (optional)

Reserve your space

Seminars in UK and Germany: The Challenges of Testing Speech Controlled Audio Systems

smart_speakersYour devices got smarter. Did your test system?

Learn how to implement open loop tests for both playback and recording in a range of devices including smart speakers, automotive audio, robots, IoT devices and more in this practical half-day seminar.

Open loop testing (testing devices where inputs and outputs are independent) enables many types of smart devices and their components to be tested in various formats and situations including:

  • Smart speakers, smart watches and other smart devices
  • Microphone arrays
  • Speech recognition systems with microphones
  • In-vehicle audio systems
  • Audio devices/systems with no physical inputs or outputs
  • Testing in noisy environments

We explain how to measure the same parameters as traditional loudspeakers, discussing such challenges as injecting and extracting response signals, time delays, and triggering the system using voice commands. We also demonstrate how to measure microphone array performance, speakerphone performance and more. Course content includes:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling

There is no charge for this one-day seminar and lunch will be provided. Space is limited, so please RSVP today.

 

Locations and Dates:

PrintUK: Tuesday Sept 25th, Sharnbrook Hotel, Bedford, England. The Sharnbrook Hotel, Park Lane (off A6), Sharnbrook Bedfordshire MK44 1LX. Note: If you need to stay here overnight, tell them you are with the ACSoft group to take advantage of a discounted rate.

 

PrintGermany: Thursday Sept 27th, NH München Ost Conference Center, Einsteinring 20, 85609 Aschheim, Germany.

 

 

Full Agenda (9am-4.00pm)

  • Introduction to Open Loop Testing
    • What is it?
    • What Kinds of Analysis Can Be Done?
      • Classic Acoustic Testing: Frequency Response and Distortion
        • Stimulus Signals
        • Distortion Methods: THD, THD+N, Rub & Buzz (Squeak & Rattle), Perceptual Rub & Buzz, Non-Coherent Distortion
    • Mixing Digital and Analog Units
    • Capturing Signals Asynchronous to the Stimulus
    • Dealing with Triggering, playback and recording Delays, digital clock Resampling, and corresponding Frequency Shift
  • Open Loop Applications for Playback
    • Smart Devices
      • Frequency Response and Distortion
      • Directionality
    • Automotive
      • 6 Mic Tree for Audio Tuning
      • Distortion methods
      • HATS for Impulse Response Testing @ seated position
  • Networking Lunch
  • Open Loop Applications for Recording
    • Smart Devices
      • Frequency Response and Distortion
      • Voice Recognition
      • Directionality
    • Automotive
      • Frequency Response and Distortion (general and to ITU P.11xx)
      • SNR placement study
      • Impulse Response function from HATS mouth -> microphone
      • Directionality
  • Handsfree Communications
    • Terminology for Handsfree Communications
    • Introduction to Telephony Metrics
  • GRAS Product Overview
    • Choosing the right microphone for your application (far field vs near field vs unconventional environments)
    • When to use a Mouth Simulator vs Head and Torso Simulator for testing

Presenters:

Steve Temme – Listen, Inc.

 

 

Seminar: Smarter Measurements for Smarter Speakers – Chicago and Boston

Smarter speakers require smarter test methods!

smart_speakersLearn how to test smart speakers, robots, voice-controlled automotive audio and other IoT devices in our practical half-day seminar taught by industry experts Steve Temme, and Marc Marroquin of Listen, Inc.

Although acoustic testing of smart speakers and other voice-activated devices presents challenges such as injecting and extracting response signals, time delays, and triggering the system using voice commands, it is still possible to measure the same parameters as traditional loudspeakers, In addition we can also measure microphone array performance, speakerphone performance and more.

We demonstrate how to do this, focusing on:

  • Stimulating and capturing responses from a device where you don’t have direct access to the microphone or speaker (open loop testing).
  • Delays, asynchronous stimulus / acquisition, and working with the ‘cloud’
  • Testing with real world signals such as speech and music, and how to analyze results from these test stimuli
  • Voice Recognition – key word spotting, wake word testing, directionality, and the effect of background noise on voice recognition performance
  • Telephony – testing smart speakers for hands-free calling

There is no charge for this one-day seminar and lunch will be provided. Space is limited, so please RSVP today.

Print

Locations and Dates:

Addison (Chicago area): Wednesday July 18th, 2018 – 9.00am – 1.30pm. Hilton Garden Inn, 551 N Swift Rd, Addison, IL 60101

Waltham (Boston area): Friday July 20th, 2018 – 9.00am – 1.30pm. Hilton Garden Inn, 450 Totten Pond Rd, Waltham, MA 02451

 

Agenda (9am-1.30pm)

  • Introduction
  • Open Loop Testing
    • What is ‘open loop’ testing?
    • Can you really measure frequency response and harmonic distortion in an open loop?
    • Using Frequency Shifting to time align asynchronous stimuli and response
    • Working with cloud based services
  • Testing with Real World Signals
    • How to use real voice and / or non test tones to evaluate acoustic performance
    • Analysis techniques used for evaluating non test tone stimuli
  • Voice Recognition
    • Key Word spotting / Wake Up word testing
    • Room effects, environmental distractors, and SNR…how they affect voice recognition performance
    • Measuring directionality
  • Telephony
    • What voice quality metrics are important for speakerphones?
    • Challenges of testing speakerphones in different environments
  • Networking lunch

Reserve your space

Smarter Measurements for Smart Speakers

Author: Daniel Knighten (Listen, Inc) and Glenn Hess (Indy Acoustic Research).  Reprinted from the March 2018 issue of Audio Xpress.

In this article, we describe techniques to characterize the frequency response, output level, and distortion of the device under test to enable direct comparisons between Internet of Things (IoT) smart speakers and conventional speakers.

Full Article