I've Been Lying to Myself About "Completion" — And My Systems Have Been Paying For It

Three months ago, I deployed a notification service that was supposed to be bulletproof. It was built on top of a job queue, had retries configured, monitoring in place, the whole checklist. Then at 2 AM, something broke during a routine deployment. The graceful shutdown didn't work cleanly, and by morning, users were getting duplicate notifications. Not a few — hundreds of them. The embarrassing part? When I dug into the logs, I realized my system was working exactly as I designed it. There was no bug. There was just my fundamental misunderstanding of what "done" actually means in a distributed system.

I've been thinking about this a lot since reading about the Trigger.dev incident. And I've come to an uncomfortable realization: I've been building systems with a naive assumption about state that no production system can afford.

The Illusion of "Done"

Here's what I thought I understood: a task completes, I mark it as done, life moves on. In practice, I've been ignoring a fundamental truth about distributed systems that's been mathematically proven since 1985.

When you process a job and that job succeeds, how does your orchestrator actually know? It waits for a signal. But what if that signal never arrives because the network hiccup happened right as the acknowledgment was being sent? From the system's perspective, the worker succeeded but never reported back. So it retries. And your system just processed the same work twice.

This isn't a bug. This is the Two Generals Problem — an impossibility proof, not a limitation we'll eventually engineer around. You cannot have both guaranteed execution and guaranteed uniqueness when failures are possible. You have to choose.

What Google and AWS Already Know (And Admit in Footnotes)

I was humbled reading Google Cloud Tasks' documentation. They're explicit: they optimize for guaranteed execution over uniqueness. Their metric of 99.999% single execution? That's still 3,650 duplicates per year at modest scale. At Google's actual scale, we're talking millions of duplicates daily.

AWS is equally frank. Their standard SQS queue delivers "at-least-once" by design. They document three specific failure modes where Lambda gets invoked multiple times for the same message. And their mitigation — storing message IDs in DynamoDB before processing — is just pushing the problem sideways. Now you have a deduplication layer that itself can fail.

The uncomfortable part: this documentation has been there the whole time. I just never read it carefully until something broke.

My Own Recent Reckoning

Looking back at my notification service, I realize I was building on a false assumption. I treated the job queue as a guarantee of exactly-once delivery, but that's not what it provides. It provides at-least-once with good retry logic.

The fix wasn't more elaborate monitoring. It was making my notification handler idempotent. If the same notification ID gets processed twice, it's a no-op on the second pass.

// Before: assumes "done" means done
async function sendNotification(job) {
  const sent = await emailService.send(job.data);
  return sent;
}

// After: assumes "done" means "done or already done"
async function sendNotification(job) {
  const { userId, notificationId } = job.data;
  
  // Check if we've already processed this
  const existing = await db.notifications.findOne({ 
    id: notificationId,
    userId 
  });
  
  if (existing) {
    return { status: 'skipped', reason: 'already_sent' };
  }
  
  const sent = await emailService.send(job.data);
  
  // Mark it processed before returning
  await db.notifications.insert({
    id: notificationId,
    userId,
    sentAt: new Date()
  });
  
  return sent;
}

The logic is simple: every piece of work gets an idempotent ID. Before I do anything, I check if I've seen that ID before. If I have, it's a no-op. This moves the guarantee from "the network will never duplicate" to "I don't care if the network duplicates because my code handles it."

What This Means for How I Build Now

I've stopped thinking about "completion" as a state and started thinking about it as a contract. The contract says: "If you give me the same work twice, I'll give you the same result without side effects."

This changes what I design for. It means I need idempotent IDs for every piece of work. It means I need to think about what "already processed" looks like in my database before I do irreversible operations. It means I need to test the failure modes that documentation warns about.

The hardest part? Accepting that my monitoring system showing "no errors" doesn't actually mean no duplicates happened. It just means they succeeded.

What I'm Still Figuring Out

I still have questions. How do you make financial transactions idempotent? How does that work across payment processors that aren't designed for it? How do you balance the cost and latency of deduplication against the cost and customer impact of duplicates?

For you, the real question is: have you tested what happens when your "done" state gets duplicated? Or are you still operating under the same illusion I was?

Source: This post was inspired by "Done" Is Not a State" by Dev.to. Read the original article