Stop Switching Tools: Why Kubernetes Operators Need to See ML Workloads Clearly

Last month, I spent three hours debugging why a Kubeflow training job was hanging. Three hours. I bounced between the Kubeflow dashboard (which told me the run was "pending"), kubectl logs (which showed nothing useful), and the actual Pod details (which revealed the real problem: an ImagePullBackOff on a sidecar I'd forgotten about). This is the kind of friction that shouldn't exist in 2026, yet it does constantly in ML-heavy Kubernetes deployments.

The deeper issue isn't that Kubeflow or similar platforms are poorly designed—they're actually quite elegant architecturally. The problem is that specialized ML dashboards and general-purpose Kubernetes tooling live in separate worlds. When things break, operators like me get caught in the middle, forced to context-switch between incompatible views of the same system. The Headlamp Kubeflow plugin finally addresses this gap, and it's worth understanding why this matters beyond just convenience.

The Gap That Exists in Production

Kubeflow is built the right way: everything is a Custom Resource Definition (CRD). Notebooks are CRDs, training jobs are CRDs, pipelines are CRDs. This is architecturally sound—it means your ML workloads follow the same Kubernetes primitives as everything else running on the cluster.

But here's what actually happens when I'm on call: the data science team uses the Kubeflow UI to submit work. When something fails, I can't just look at that same UI to see what went wrong. I have to drop into kubectl, navigate Pod conditions, check volume mounts, examine node tolerations, and piece together what the actual Kubernetes layer is reporting. The specialized dashboard hides all that infrastructure detail, which is great for scientists but terrible for operators trying to troubleshoot.

The Headlamp plugin flips this. Instead of forcing operators to leave their Kubernetes tooling, it brings Kubeflow's ML-specific resources directly into the general-purpose UI. It's not revolutionary—it's just solving a real pain point that anyone operating Kubeflow at scale has experienced.

What This Plugin Actually Gives You

The plugin supports the major Kubeflow components: Notebooks, Pipelines, Katib (hyperparameter tuning), distributed training jobs, and Spark applications. For each, it surfaces the Kubernetes truth directly from the API server.

For notebooks, you get Pod conditions with actual failure reasons—not abstract dashboard statuses. For pipeline runs, instead of querying some backend database, it reads the CRDs directly. This is significant because it means you can diagnose issues even when the ML backend is broken. For Katib experiments, you see the full trial lifecycle with early-stopping metrics. For training jobs, you get visibility into resource requests, volume mounts, environment variables, and node scheduling constraints.

The plugin also includes a graph view that maps relationships between resources using ownership references. This helps answer questions like "which pods actually belong to this notebook?" without needing to mentally parse kubectl output.

My Take: Smart Pattern, But Limited Scope

I like this approach. It acknowledges that operators need a different view than users, and rather than building a completely separate tool, it extends an existing Kubernetes UI. That's pragmatic.

But I have questions. The plugin discovers which Kubeflow components are installed and only shows relevant sections. That's good UX. However, what about custom CRDs that teams build on top of Kubeflow? Our organization has internal training abstractions that wrap Kubeflow's TrainJob. The plugin won't automatically understand those. You'd need to write another plugin, which adds complexity.

Also, this assumes your team already uses Headlamp or is willing to adopt it. For orgs deeply invested in Lens, Octant, or custom dashboards, switching cost might be real.

The pattern the article highlights is valuable though: meet operators where they already work, and show cluster-level truth. That's a principle I'd apply broadly.

The Practical Win

Here's what I'd do immediately if I deployed this: use it during incident response. Instead of jumping to three different tools, I'd have a single place to inspect Kubeflow workloads and their underlying Pod state. That alone saves time and reduces cognitive load when things are broken and stress is high.

For ongoing operations, having direct API visibility (not going through a backend service) is genuinely useful. If your Kubeflow Pipelines backend is having issues, you can still diagnose pipeline state from the CRDs.

The resource mapping visualization would help new team members understand how notebooks, experiments, and training jobs relate to each other—something that's not obvious from either the Kubeflow UI or raw kubectl.

What Would You Do Differently?

If you're running Kubeflow today, what's your current debugging workflow? Are you also bouncing between dashboards and kubectl, or have you found a better pattern? I'm curious whether the separation between specialized ML UIs and general Kubernetes tooling feels like the right design to you, or whether something like Headlamp changes that perspective.

Source: This post was inspired by "Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow" by Kubernetes Blog. Read the original article