Why I'm Skeptical About Using ML for Real-Time Anomaly Detection (And What I'd Do Instead)

Last month, I got pulled into a security meeting where someone suggested we use machine learning to detect "unusual sounds" in our office for safety monitoring. The pitch was slick: train a model, let it run 24/7, catch problems before they happen. I nodded along, but internally I was already thinking about failure modes. A few days later, I stumbled on research testing whether YAMNet—Google's audio classification model—could actually detect unseen sounds in real time. The results? 45.8% recall. That number haunted me because it confirmed what I've learned the hard way: ML models are often sold as magic bullets when they're actually quite fragile.

This isn't a criticism of the research. It's rigorous work. But it's also a reality check for anyone considering deploying audio anomaly detection in production. The gap between what sounds possible and what actually works is enormous.

What YAMNet Actually Is (And Isn't)

YAMNet is a lightweight audio classification model built on MobileNet that Google released in 2019. It processes ~1 second of audio at a time and outputs two things: predictions across 521 sound classes and a 1,024-dimensional embedding—basically a compressed numerical representation of what the audio "sounds like."

The appeal is obvious. It's pre-trained, relatively fast, and handles a massive range of acoustic events. But—and this is critical—it was trained to recognize known sounds. The question the researcher tested was fundamentally different: can it detect when something unexpected happens, even if that thing isn't in its training data?

The experiment was clever. They mixed in sudden events (glass breaking, sirens, car horns) into normal background audio (rain, wind) at varying volumes, then checked whether the model could flag that something changed without being told what to look for. The best detector caught 22 out of 48 events.

The Problem With Distance Metrics in Embedding Space

Here's what got me thinking: the researcher tested two approaches. The first measured distance from the previous sound frame (delta detection). The second compared against a memory of "normal" sounds using k-nearest neighbors.

Delta detection won. Embedding delta AUROC was 0.734, which is better than random but not trustworthy enough for production alerting.

I think the reason is instructive. Sound is continuous. Rain flows into wind. Wind carries occasional ambient noise. A sudden car horn breaks that pattern. You don't need a complex embedding comparison to notice disruption—you just need to measure change. But here's the catch: measuring change is trivial. Building something that only alerts on meaningful change while ignoring normal variation is the hard part.

The researcher found that embedding distance performed marginally better than score distance, but we're talking 0.734 vs 0.717. That's not a meaningful difference across 80 samples. It tells me that throwing a fancier representation at the problem doesn't solve the fundamental issue: novelty detection in audio is genuinely hard.

What I'd Actually Build Instead

If I were tasked with detecting unexpected sounds in a real office, I wouldn't start with YAMNet embeddings. I'd start much simpler.

Spectral flux—basically measuring frequency changes frame-by-frame—got AUROC of only 0.449, which sounds terrible. But it's honest. It's also extremely fast and interpretable. I know exactly why it fired. I can tune it without touching a GPU.

My actual strategy: layer simple detectors. Spectral flux for sudden energy changes. Peak detection for sharp acoustic transients. Silence breaks. These run instantly on a CPU. Then, if I have time and budget, add YAMNet as a secondary classifier to rule out false positives. "Yes, the sound changed suddenly AND YAMNet thinks it might be a door slamming, so maybe alert."

The researcher's conclusion that "a simple distance over YAMNet embeddings cannot reliably detect unseen sudden sounds" is honestly the most valuable finding here. It's a no. Not "more research needed." Not "promising but needs tuning." A straightforward no.

My Real Concern: Precision Matters in Production

The precision/recall table shows the tradeoff clearly. At threshold settings that produced zero false alerts during calibration, score delta achieved 45.8% recall. That means if an actual event happens, you'll miss it more than half the time.

In real production systems, this is unacceptable. You either accept high false alarm rates (and users ignore alerts) or low recall (and you miss actual incidents). There's no magic threshold that solves this.

I've shipped audio-based systems before. The lesson I learned: if you can't get to 90%+ precision with your main detector, add more domain constraints. Narrow the problem. Don't try to detect "anything unusual"—detect "glass breaking near the east window during business hours." Constraints are your friend.

What's Next

I'm genuinely interested in whether temporal fusion or more sophisticated anomaly detection methods would help. But I'm skeptical that the answer is "better ML." It might be better signal processing combined with selective ML.

If you're considering audio anomaly detection for a real project, read the original research carefully. The honest assessment of what doesn't work is more valuable than any inflated success rate.

Source: This post was inspired by "Can YAMNet Detect Unseen Sudden Sounds in Real Time? A 48-Stream Evaluation" by Dev.to. Read the original article