Stop Inferring State From Your Logs—Pick a Marker Instead

I was debugging a production issue at 2 AM last month when I realized I'd been doing something stupid. Our system was showing the wrong status for batch processing jobs because I'd built state inference on top of scattered log lines instead of a single, intentional marker. I kept thinking: "why is this so fragile?" Reading about someone else's three-round journey through the same hell was oddly comforting.

The original problem sounds innocent enough—show users which site is being maintained right now, which ones are done. Standard UI pattern. But I've learned the hard way that streaming logs are a terrible source of truth for state inference if you're not deliberate about it. The article walks through this exact lesson, and it's worth internalizing because I think most developers miss it the first time around.

The Trap: Inferring State From Everything

The initial approach was seductive in its simplicity. Watch the logs. When a new site name appears, the previous one must be done. Mark it complete. Move on.

I've done this exact thing. You're reading logs, you see a change, you assume that change means something happened. It's a natural instinct. But logs aren't ordered the way you think they are—especially in streaming scenarios where initialization happens before processing, where buffering creates non-monotonic ordering, where multiple components emit data simultaneously.

The real problem emerged during testing: sites marked as "done" before they'd even started. The system saw Site B in an initialization log, then Site A starting for real, and concluded "oh, A is running now, so B must be done." Nope. B hadn't started yet.

The First Fix That Didn't Actually Work

Adding an execution-order index helped with the rewind problem. By tracking "what order should these sites run in" and ignoring any log that jumps backward, they plugged that specific hole. It felt better, more structured. But it was treating a symptom, not the disease.

The real issue was that the inference logic was fishing for state in the wrong places. Pre-emitted initialization logs—[Site B] Injecting connection info—were being parsed as "site B is running." The order-index fix couldn't save you from that because the problem wasn't disorder; it was wrong data.

The Actual Solution: One Clear Marker

This is where the article nails it. Stop inferring. Pick one line type that means exactly one thing: [Site A] Starting maintenance. That's it. Only watch that line. Every other log is noise from your perspective.

I found myself nodding hard at this. In my own work, I've started applying the same principle. Instead of building inference logic that reacts to multiple signal types, I create explicit markers. A single event type that has no ambiguity. A single source of truth for state transitions.

The code shift is small but conceptually important:

// BEFORE: Fish for state everywhere
function _inferSite(lines) {
  // "did the site name change?" check across all lines
  // Fragile against non-monotonic ordering and noise
}

// AFTER: Watch for one marker only
const TRANSITION_MARKER = /Starting maintenance/;

function _detectRunningSiteFromLog(lines) {
  for (let i = lines.length - 1; i >= 0; i--) {
    if (!TRANSITION_MARKER.test(lines[i])) continue;
    // Only process this line
    const site = extractBracketName(lines[i]);
    _setRunningSiteId(site);
    return;
  }
}

The difference is philosophical: you're not trying to be clever. You're not pattern-matching across contexts. You're saying "this one line type means this one state transition, full stop."

What I'd Add: Separate Inference From Persistence

One aspect the article mentions but I think deserves emphasis: don't mark completion at the same time you infer what's running. The original approach marked the previous site done when detecting a new site. That couples your inference logic to your state management, which means every inference mistake becomes a persistence mistake.

They moved completion marking to the end of the run, in batch. That's wise. It means inference jitter doesn't cascade into wrong state. Your UI might flicker about what's running, but at least completion marks are stable and backed by actual execution results, not by inference timing.

I've started doing the same thing in my own projects—keeping inference and persistence separate layers. Inference can be noisy and exploratory. Persistence should be deliberate and backed by clear events.

The Scope Problem I Keep Forgetting

The third bug was smaller but illuminating: clearing all "completed" status instead of just the sites in the current run. That's the kind of bug that feels embarrassing because the fix is obvious in retrospect. But I've shipped code with that exact pattern—"clear everything" instead of "clear the right subset."

It's a reminder that even cleanup operations need to be scoped. State management isn't just about setting things; it's about knowing what not to touch.

What This Means for Your Streaming System

If you're building anything that infers state from logs or events, start with your markers. What is the one line or one event that unmistakably means "thing X happened"? Build around that. Ignore everything else. Keep that marker stable and well-defined.

Stop being clever about state inference. Pick one source of truth and stick to it.

Source: This post was inspired by "Detecting the running site from streaming logs — why log-order inference broke, and how one marker fixed it" by Dev.to. Read the original article