Stop Blaming Your Network: Why Kafka Cross-VPC Failures Are Actually an Addressing Crisis

I spent three hours last month debugging what looked like a textbook networking failure. A Kafka consumer in one AWS VPC couldn't pull messages from a cluster in another. Telnet worked. Security groups were open. The routing table looked clean. Everything pointed to "network is broken."

It wasn't. And I felt stupid when I realized the actual problem.

The issue was that I was thinking about Kafka like it was a normal service—throw a load balancer in front, slap one DNS name on it, done. Kafka laughs at that approach. This realization changed how I approach multi-VPC architectures entirely, and I want to break down why this distinction matters.

How Kafka Actually Connects (And Why It's Different)

Most services have a simple connection pattern: client → load balancer → service. Kafka breaks this into two stages, and the second stage is where everything falls apart across network boundaries.

When a Kafka client first connects, it performs a bootstrap handshake with any broker, asking for cluster metadata. The broker responds with a list of all brokers and their advertised addresses. Here's the critical part: the client then opens direct connections to specific brokers based on partition leadership. It's not connecting through some central proxy—it's routing directly to addresses the broker told it to use.

If a broker advertises broker-1.internal:9092 (a private hostname that only resolves inside its VPC), a remote client can bootstrap fine but immediately fails when trying to reach that specific broker from a different network. The network path exists, but the address resolution fails. That's an addressing problem masquerading as a networking problem.

Why Standard Solutions Fail

I initially thought VPC peering would solve this elegantly. Two VPCs, one peering relationship, problem solved. That worked until we added a third client network.

VPC peering is point-to-point and non-transitive. If I peer Network A to the Kafka cluster's VPC and Network B to the cluster's VPC, A still can't talk to B without a separate peering. Scale this to a dozen client networks across different AWS accounts and regions, and your infrastructure becomes a nightmare of bidirectional peerings, CIDR coordination nightmares, and constant VPC modifications.

Transit gateways initially seemed like the answer—one hub that many VPCs attach to. But they operate at Layer 3 and don't understand Kafka's protocol. They move IP packets blindly and can't rewrite the broker addresses in metadata responses. You're still stuck with unreachable private hostnames.

Per-broker load balancers create their own problems. You'd need an NLB per broker, each with its own DNS record and TLS certificate SAN. Any cluster scaling becomes a rolling restart nightmare to keep the advertised addresses in sync.

The Real Solution: Protocol-Aware Proxying

The elegance of a Kafka-aware proxy hit me immediately. Instead of trying to hack networking, you insert something that understands the Kafka protocol and rewrites metadata before the client sees it.

A proxy sits in a hub VPC, understands the bootstrap-then-direct-connect pattern, and rewrites broker addresses in metadata responses. Each broker gets its own port on the proxy, preserving deterministic routing. The client changes one line of configuration:

# Before: pointing at unreachable broker
bootstrap.servers=broker-1.internal:9092

# After: pointing at the proxy
bootstrap.servers=kafka-gateway.hub:9092

The proxy is stateless and credential-agnostic—it just forwards SASL credentials through to the brokers. Your existing ACLs or RBAC policies continue to work unchanged.

My Take

This approach is architecturally sound, but I have questions about operational complexity. Proxies add another moving part, another thing to monitor, scale, and troubleshoot. In high-throughput scenarios, I wonder about latency impact and how performance degrades under load.

I also think this reveals a fundamental issue with how Kafka was designed for distributed systems. Its two-stage connection pattern made sense for on-premises clusters in single networks, but it doesn't scale gracefully in modern cloud architectures where you're routinely crossing network boundaries.

For most teams, especially those on managed Kafka services (MSK, Confluent Cloud), this gateway pattern seems like the pragmatic path forward. But I'd want to thoroughly test performance and failure modes before deploying it to production.

The real takeaway for me: always distinguish between network problems and address resolution problems. They look similar from the outside but need completely different solutions.

What Would You Do?

Have you dealt with multi-VPC Kafka setups? How did you solve the addressing problem, and what was your experience with proxy-based solutions versus other approaches? I'm genuinely curious whether the operational overhead is worth it at scale.

Source: This post was inspired by "Kafka addressing & VPC peering" by Dev.to. Read the original article