Why I Finally Stopped Fighting Kubernetes Autoscaling and Built a Custom Metrics Exporter

I spent three months last year watching a Kubernetes cluster make the worst scaling decisions I've ever seen. The HPA would spin up ten new pods because CPU spiked for thirty seconds, then crash them all when traffic dropped. Meanwhile, our actual bottleneck—a Redis queue growing to 50,000 items deep—went completely invisible. CPU and memory metrics told me nothing about whether our system was actually healthy. That's when I realized: Kubernetes's built-in autoscaling is blind to what actually matters in your application.

The turning point came during a 2 AM incident where we had plenty of CPU headroom but jobs were timing out because they were starving in the queue. I decided that night to stop living with this limitation and actually build something better. A custom metrics exporter turned out to be simpler than I expected, but it forced me to think clearly about what signals actually drive my scaling decisions. That clarity alone was worth the effort.

Understanding What an Exporter Actually Does

An exporter is fundamentally simple: it's a tiny HTTP server that turns internal application state into numbers that Prometheus can understand. Nothing more. Your exporter doesn't need to be clever—it just needs to be honest.

The magic happens because Prometheus scrapes that /metrics endpoint on a schedule, stores the time-series data, and makes it available for the HorizontalPodAutoscaler to make scaling decisions. But here's the thing that clicked for me: you're not limited to CPU and memory anymore. You can expose anything that matters. Queue depth, active WebSocket connections, database pool saturation, the age of the oldest pending task—whatever drives your actual system behavior.

I've seen teams try to instrument everything directly in their application code. Sometimes that works, but it gets messy. A standalone exporter keeps concerns separated: your application focuses on its job, and your exporter focuses on observation. This also means you can update your metrics strategy without redeploying your entire service.

Choosing What Actually Matters

Before I wrote a single line of code, I had to think hard about what to measure. The Prometheus data model gives you three basic types, and picking the right one matters more than you'd think.

Counters only go up. Use them for totals: jobs processed, errors, requests served. Never for something that can decrease.

Gauges are snapshots of the current moment. Queue depth, active connections, cache size. These are usually what you want for autoscaling decisions.

Histograms capture distributions. Request latency percentiles. Useful for understanding performance, less useful for autoscaling.

I made a mistake early on trying to expose everything as gauges. I should have been more intentional. For our worker system, I needed: worker_queue_depth (gauge—current queue size), worker_jobs_processed_total (counter—lifetime total), and worker_job_duration_seconds (histogram—for monitoring, not scaling). These three metrics told the complete story.

What This Means in Practice

Here's the thing about implementing this: it's straightforward, but discipline matters. Go's Prometheus client library handles all the serialization, so you're really just deciding what to measure and calling the right function. The registration pattern—using prometheus.MustRegister—is brilliant because panicking on duplicate registration forces you to catch misconfigurations at startup, not silently in production.

The polling loop pattern in the original article is solid, but I've learned you need to think about your scrape interval. If Prometheus scrapes every 15 seconds and you're updating metrics every 5 seconds, you're fine. But if you're polling faster than Prometheus scrapes, you're doing unnecessary work. If you're polling slower, you're missing freshness. I usually set my collection interval to about one-third of my Prometheus scrape interval.

// This is the core pattern I use
var queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
    Name: "worker_queue_depth",
    Help: "Current jobs in queue",
})

func init() {
    prometheus.MustRegister(queueDepth)
}

func collectMetrics() {
    ticker := time.NewTicker(5 * time.Second)
    for range ticker.C {
        depth := getQueueDepth() // Your actual data source
        queueDepth.Set(float64(depth))
    }
}

func main() {
    go collectMetrics()
    http.Handle("/metrics", promhttp.Handler())
    http.Handle("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    http.ListenAndServe(":8080", nil)
}

The /healthz endpoint is a nice touch that I borrowed immediately. It lets Kubernetes probe your exporter's liveness without scraping metrics, keeping concerns separate.

My Honest Take

This approach works, and it's not overengineered. But here's what I'd push back on: the original article treats custom exporters as the obvious solution, when the real question should come earlier. Before building an exporter, ask whether you should instrument your application directly instead. If you own the code and the metric is central to your service's behavior, embedding the Prometheus client might be simpler.

That said, an external exporter shines when you're reading from systems you don't control—message queues, external APIs, databases. That's where I've found the most value.

What's Your Bottleneck?

The uncomfortable question this raises: do you actually know what's driving your scaling needs? Most teams I've worked with scale on CPU because it's easy to measure, not because it's the right signal. Before you build an exporter, spend time understanding what actually makes your system healthy or sick. That clarity is worth more than the code.

Source: This post was inspired by "Building a Custom Metrics Exporter for Kubernetes" by Kubernetes Blog. Read the original article