The Instinct That Almost Cost Me Everything: Why Your First Move Shouldn't Be a Hardware Replacement

I was debugging a storage issue at 2 AM last year when my senior ops person told me to "just replace the disk." I was about to. The metrics looked terrible — multiple devices showing as unreachable, the whole thing smelled like hardware failure. But something made me pause. I ran a quick network trace, found a misconfigured switch port, fixed it, and watched everything come back online without touching a single piece of hardware. That single decision saved us probably six hours of work and a lot of unnecessary component swaps. Reading about someone else hitting the exact same pattern with ZFS made me realize this isn't a coincidence — it's a principle I'd figured out empirically, and it applies everywhere in infrastructure work.

The article that sparked this for me walks through recovering a suspended TrueNAS pool that looked genuinely dead. Eleven disks, multiple faults, a removed device, the whole grim picture. The instinct — the reflex — would be to start pulling hardware and replacing disks. But the author dug deeper first. That's the mindset I want to talk about today, because I think it separates people who fix problems quickly from people who create new ones while trying to solve existing problems.

When Multiple Failures Cluster, Look For Shared Infrastructure

Here's what I've internalized from years of dealing with distributed systems: simultaneous failures rarely happen independently. When three drives fail at once, when five network interfaces drop at the same time, when a whole rack of VMs goes down together — they usually share something. A power supply. A cable. A controller. A firmware update that just went out. A network misconfiguration. Something in the path, not the endpoints.

The recovery approach in the article is built on this principle. Multiple devices showing as faulted at the same time isn't a drive reliability issue — it's a signal that they're all connected through something that failed. That's a completely different diagnostic path than checking individual drive health.

The Dangerous Reflex: Do Nothing First

What grabbed me most about this article is the explicit list of things not to do. Don't add disks. Don't wipe drives. Don't replace anything. Don't start a scrub. I've seen developers — good ones — trash storage arrays by rushing to "fix" things in the wrong order. The muscle memory when something looks broken is to immediately act on it. That's backwards when you're dealing with interconnected systems where the wrong action can make things unrecoverable.

The author talks about scrubs being a validation tool, not a recovery tool. I hadn't thought about it explicitly that way before, but it's correct. Running diagnostics on a system that's still in a bad state doesn't help you recover — it just hammers the broken path harder. First stabilize, then validate.

What Actually Worked: Diagnostic Before Action

The recovery steps here are methodical in a way that feels almost boring until you realize how many people skip them:

Can the OS see the disks at all? This is the free diagnostic that tells you which class of problem you're in — hardware, firmware, driver, or software.
Are there active errors in the kernel logs? This tells you whether the path is currently stable or still blowing up. If it's still blowing up, you fix the path before you touch the software layer.
Only then run the recovery command that the system suggested (zpool clear in this case).

The elegance here is that each step costs almost nothing — a few seconds of commands — and each one narrows down the problem space dramatically. By the time you run the recovery command, you already know it has a reasonable chance of working.

My Take: The Pattern Applies Everywhere

I work with APIs, databases, and microservices. The principle is the same across all of it. When something looks catastrophically broken, before you:

Redeploy containers
Clear caches
Restart services
Rebuild databases
Reindex everything

...ask: what do these failures share that isn't the thing I think is broken? Is it a network issue? A dependency? A recent deployment? A quota? A permissions change?

The difference between the person who solves a problem in ten minutes and the person who takes two hours is usually the order in which they investigate. The fastest way to make a bad situation worse is to start replacing things before you understand what went wrong.

The other thing this reinforced for me is that being verbose about failure states matters. The kernel logs, the zpool status output, the smartctl scans — all of these are communication from your system about what actually happened. People who learn to read that language faster get better at this work.

What Would You Do?

If you hit something like this in your infrastructure — whether it's storage, databases, or anything else that looks catastrophically broken — what's your first move? I'm genuinely curious whether people are building this diagnostic-first muscle memory or if the "turn it off and turn it back on" reflex still dominates.

Source: This post was inspired by "How I got a suspended TrueNAS pool back online without replacing a single disk" by Dev.to. Read the original article