The Value of Big Data

There’s been a lot of recent discussion about AI in audio test and measurement and some companies in our industry are openly talking about using data collected from customer production lines to train their models. The premise is simple: the more real-world manufacturing data you have, the better your algorithms will become. This raises 2 big questions: 

• Is the data truly valuable?
• Is this ethical?

For me, the answer is “Rarely” and “No”. Let me explain.

Is More Really Better?

A database with ten million loudspeaker measurements is an intriguing proposition, but I’d have a lot of questions before I relied on any of that information to develop an algorithm. What products were being measured? What measurement hardware was used? What were the precise test parameters – the stimulus signal, the specific algorithm used, etc.? What was the background noise level? Do we have the raw time recordings? Are there subjective listening results so we  know what an experienced listener actually heard? And even with answers to all of those questions, I would still need to trust that the test was performed competently and repeatably with a calibrated system and sufficient signal-to-noise ratio.

Making good audio measurements is never as simple as just connecting a microphone and pressing a button. This is particularly true on a production line, where there is constant pressure to maximize yield and throughput, and operators sometimes find very creative solutions to the problem immediately in front of them.

Once, at a customer’s production facility, an operator showed me just how easy it was to get a speaker with bad Rub & Buzz to pass the end-of-line test. He simply opened the test box and introduced background noise during the measurement. The noise masked the higher-order harmonic distortion, and suddenly the failing speaker passed the test. The yield went up, but of course the manufacturer’s reputation for quality didn’t! Now imagine that result being automatically added to a large dataset and subsequently used to train or refine an algorithm. The data says the speaker passed, but anyone who observed the measurement knows the data is worthless.

It’s not always that intentional. Recently, someone challenged a measurement I made as it didn’t agree with the manufacturer’s data or previous measurements made by a well-known third party. We had three different results for the same speaker, all made by highly qualified, experienced engineers using different test equipment, and mine was the outlier. A deep dive into the test conditions for each measurement revealed the source of the discrepancy: the measurement space. When the speaker was re-measured outdoors, many feet in the air in the middle of a large field, to approximate a free-field environment, the results demonstrated that SoundCheck’s nearfield/farfield “splice” technique was accurate and that limitations in the other measurement setups accounted for the discrepancies.

If I had been handed those three sets of results without knowing how they were obtained, which one should I have treated as correct? And if all three had been anonymously added to a huge database, what would an AI model have learned from them?

This is the fundamental problem with the idea that simply harvesting vast quantities of real-world measurement data will produce better algorithms. Calibration, equipment, environment, test configuration and operator behavior all affect the result. Without that context, you might have a lot of numbers, but it’s not a valuable dataset.

The Value of Trusted Data

This doesn’t mean real-world customer data isn’t useful. Some of our most successful algorithms have been developed with it. The difference is that we know what the data represents.

For example, our perceptual distortion algorithms were developed using customer-supplied defective speakers together with subjective listening results. We could measure the same devices repeatedly under controlled conditions, compare the results with what experienced listeners actually heard, and experiment with different analysis techniques. The value wasn’t the quantity of data; it was knowing exactly what we were measuring and having the context to interpret the results.

The same principle applies if AI becomes part of audio test and measurement. Our measurement algorithms don’t currently rely on AI or massive datasets, although I can certainly see applications for AI in the future, such as intelligent production limit setting. If we go down that road, I would rather start with a hundred measurements I understand and trust than millions I don’t. “Garbage in, garbage out” has never been more relevant.

Whose Data Is It Anyway?

Even if indiscriminately collected production data were enormously valuable, it’s not ours to collect.  Our customers use SoundCheck to develop unreleased products and manufacture millions of devices. Their measurement data can reveal not only technical details about their products, but commercially sensitive information such as production volumes and manufacturing relationships.

We welcome customers sharing data or defective products with us, under NDA if requested, as this is invaluable for developing our algorithms. However, there are two important distinctions: 1) the customer is choosing to give us that data and knows exactly what they have shared, and 2) because we know where it came from, we can ask questions to understand the context behind the measurements.

This is very different from viewing an installed customer base as a source of training data – an approach I have seen proposed by one company in our industry. They clearly see customer data as valuable for improving their algorithms and gaining a competitive advantage, but that doesn’t make it theirs to take. 

Transparency is important. If a test equipment vendor is collecting customer data, their customers should know exactly what is being collected, where it goes, and how it will be used. And if that data is being used to develop future commercial products or algorithms – which may then be sold to their competitors – there should be no ambiguity about that either.

When was the last time you checked your test equipment vendor’s privacy and data collection policy?

Your Data, Your AI

Perhaps there is a better approach? Rather than measurement companies collecting data from thousands of customers to develop generalized AI models, I believe some of the most valuable applications of AI will come from customers using their own data. After all, you know exactly what product was tested, how it was measured and what constitutes an acceptable or unacceptable device. Measurement results can be combined with proprietary information such as manufacturing parameters, component changes, failure analysis, warranty returns and subjective evaluations. That makes your own carefully curated dataset infinitely more valuable than millions of unrelated measurements collected elsewhere.

SoundCheck facilitates this. Test results can be saved with relevant metadata and exchanged with external applications via TCP/IP. This enables you to use your SoundCheck measurements with your own AI tools, while keeping your data and intellectual property under your control.

And to be absolutely clear, SoundCheck does not secretly collect your measurement data. It does not send your measurements, test results, production statistics or other information back to Listen, and there is no telemetry running in the background. SoundCheck functions perfectly well offline, as it always has. In fact, many production lines, particularly in China, aren’t connected to the internet at all.

This is a deliberate choice. We believe our role is to give customers accurate, trustworthy measurements, preserve the context that makes those measurements meaningful, and provide the tools to use that data however they choose – including with their own AI.