The Default That Almost Broke My API: Why Choosing How to Fail Matters More Than You Think

Three years into building production systems, I almost shipped a bug that would've been worse than any security vulnerability I've actually caught. It wasn't a SQL injection or a weak password hash — it was something quieter and more dangerous: a catch block that said return true when something went wrong.

I was building an internal API gateway for a client, and when their auth service hiccupped during load testing, my gateway just... kept serving requests. Every single one went through. I only noticed because I was paranoid enough to check the logs. That moment taught me something I wish I'd understood earlier: there's a security decision buried in every error handler, and most developers make it without even realizing it.

What Happens When Things Break?

Let me be direct: every system will fail. Networks drop, databases go down, services timeout. The question isn't if something breaks — it's what your code does when it does.

I'm talking about the choice between two modes of degradation. When your authentication service can't respond, does your app keep letting people in? That's failing open — prioritizing availability. Or does it lock everything down until you fix the problem? That's failing closed — prioritizing security.

The door analogy from the original article stuck with me because it's genuinely how I think about this now. A fire exit fails open because dead people don't care about access control. A bank vault fails closed because an open vault defeats the purpose entirely. The context determines which failure mode is acceptable.

Where I've Seen This Mess Up Real Systems

In my current work, I'm dealing with feature flags and configuration services that go down with surprising frequency. Early in one project, I wrote code that failed open on flag retrieval: if the service was down, every flag defaulted to true. Seemed reasonable until a beta feature started showing up in production because the config service had network issues. It was only 15 minutes, but it was 15 minutes too long.

The other extreme bit me once too. I over-corrected on a payment processing system and made everything fail closed when the policy service timed out. That meant legitimate transactions got queued indefinitely, customers got frustrated, and our support team spent hours sorting out what happened. Both approaches can hurt, but they hurt differently.

The Real Principle

The article gets at something crucial that I've started applying to every new system: the sensitivity of the operation determines how it should fail.

High-sensitivity operations — anything involving authentication, authorization, payments, or data modification — should fail closed. If your permission system is down, the default must be "deny." This is non-negotiable. I now write explicit guards:

async function checkUserCanDelete(userId, resourceId) {
  try {
    const hasPermission = await permissionService.verify(userId, 'delete', resourceId);
    if (!hasPermission) return false;
    return true;
  } catch (error) {
    // Permission check failed - we can't verify, so we deny
    logger.error('Permission check failed - denying operation', { userId, resourceId, error });
    return false; // EXPLICIT fail-closed
  }
}

Low-sensitivity operations — logging, metrics, notifications, analytics — should fail open. If my observability pipeline goes down, I'm not blocking the whole application. That's just poor engineering.

My Take

What frustrates me is that this isn't taught as seriously as it should be. We learn about error handling, but we don't learn about intentional error handling. Most developers just catch exceptions and either log them or silently ignore them. Neither is a decision — it's just whatever seemed easiest that day.

I started adding security reviews to my error handling specifically because of this. Before I write a catch block, I ask: "If this fails, what's the worst thing that could happen?" If the worst thing is "attackers get in," it fails closed. If the worst thing is "we lose some metrics," it fails open.

The article mentions this applies especially to AI-native systems, and I think that's important. As we push more autonomous systems into production, this question becomes even sharper. An AI agent that fails open on access control is genuinely dangerous.

What I'm Doing Differently

I'm now explicitly documenting the fail-mode of every critical service boundary. Not in comments — in actual code and tests. I write tests that verify what happens when a service times out or returns an error. It seems tedious until it catches something important.

How are you handling this in your code? Are you choosing your failure modes deliberately, or are you discovering them during incidents?

Source: This post was inspired by "Fail-open vs fail-closed: the security decision you make without realizing it" by Dev.to. Read the original article