The Failure Mode Nobody's Testing For: Why AWS's Zonal Shift Changed How I Think About Resilience

I was on-call at 2am last month when something weird happened. Our error rate spiked, but only for one service. The zone that hosted it wasn't down—it was just… slow. The health checks passed. The circuit breakers didn't trip. So we didn't do anything, because the alarm thresholds were tuned for actual failures, not partial degradation. By the time we realized what was happening, we'd been serving degraded traffic to thousands of users for forty minutes.

That's the failure mode AWS is finally naming as its own thing: gray zone failures. Not red, not green. And if you're running EKS in production, the way AWS is handling this should change your deployment safety checks.

What's Actually Happening Here

AWS introduced zonal shift—a mechanism that automatically redirects traffic away from zones showing signs of degradation, even when traditional alarms aren't firing. The key insight from their fleet-wide data is brutal: the failures that cause the longest outages aren't the ones that look broken. They're the ones that look slow.

A zone can lose 20% of its capacity and still answer health checks. Latency creeps up on some request paths while others stay clean. Your dashboards show scattered P99 spikes instead of a clear incident. So nobody pages, traffic keeps flowing there, and your partial outage stretches on indefinitely.

AWS's solution is to let their own automated detection—built on patterns they've seen across their entire customer base—shift traffic away from these zones before human operators even notice. The burden moves from "someone has to spot this and act" to "did AWS's detection make a mistake?"

The Deployment Problem You're Not Ready For

Here's what kept me up thinking about this: if AWS can automatically shift traffic during my canary rollout, my release monitoring becomes incomplete.

I'm watching error rates climb in my canary, thinking my new code is broken, so I rollback. But the real culprit was a zone shift that AWS fired three minutes into the deploy. My rollback was the wrong move. I just destroyed confidence in a release that was actually fine.

This isn't theoretical. It happens when your deploy tools and your traffic control tools don't see the same world. Your CI/CD pipeline needs to know whether active zone shifts are happening in the regions where your canaries are running.

Testing the Failure Mode You've Been Skipping

Most teams run two chaos scenarios: node failures and zone failures. Total outages. Easy to simulate, easy to detect, easy to recover from.

What I've started adding to our resilience testing is the scenario AWS is solving for: a zone that keeps responding, but slowly. No network partitions. No cascading failures. Just latency creeping up everywhere.

Here's what that test looks like in practice:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: gray-zone-simulation
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      zone: us-east-1a
  delay:
    latency: "150ms"
    jitter: "50ms"
  duration: 5m
  scheduler:
    cron: "@hourly"

This isn't killing anything. It's making one zone slower. Then you watch what happens. Does your service degrade gracefully? Do your rate limits protect downstream services? Does your monitoring pick it up? Most importantly: does your deployment controller know this is happening and adjust its canary promotion criteria?

My Take: This Shifts Responsibility, Not Risk

I like that AWS is automating away the "spot the gray failure" detection—that's legitimately hard at scale. What I'm skeptical about is the hidden tuning parameters. When does AWS consider a zone "degraded" enough to shift? What's the propagation delay? Can I override it when their heuristics disagree with my workload?

The article doesn't detail this, which is exactly the kind of opacity that causes problems at 3am. You need to know these numbers. You need to test with them. You need to document what your own detection looks like independently.

I'm also not convinced that fleet-wide patterns are the right signal for every workload. A pattern that AWS sees across millions of customers might not match my specific latency-sensitive request paths. I'd rather have AWS's detection as one input to my decision tree, not the decision tree itself.

What I'm Doing Next

I'm adding a section to our runbooks specifically for "deploy in progress + active zonal shift." The decision logic needs to be explicit: if a zone shift fired during canary, do we pause the rollout? Do we widen the sample size before promoting?

I'm also instrumenting visibility into when shifts happen so we can correlate them with our own metrics. AWS has one signal. We need ours.

What failure modes are you testing for that your chaos tooling doesn't cover yet?

Source: This post was inspired by "AWS puts gray zone failures into the EKS control loop" by Dev.to. Read the original article