Stop Shipping Skills on Vibes—You Need Actual Metrics

Last month, I spent three days building what I thought was a killer Claude skill for handling API documentation generation. It looked good. The outputs felt right. I shipped it to production feeling genuinely confident. Two weeks later, I noticed it was making things worse in edge cases—specifically for GraphQL schemas—but I had no way to prove when the regression started or how badly it was broken.

That's the moment I realized I've been running blind. We all have.

The Problem We're Not Talking About

Here's what gets me: everyone building with Claude has written a SKILL.md file. Almost nobody can actually say whether it works. We eyeball a few test cases, get a warm feeling, and ship it. I've done this exact dance more times than I'd like to admit. The honest answer to "does this skill actually improve Claude's outputs?" for most teams is "probably? maybe? vibes say yes."

That's not evaluation. That's gambling with system prompts.

The worst part is we have three very real failure modes nobody discusses. Position bias means when Claude evaluates its own work, it favors whatever output it sees first—so your "before and after" comparison is meaningless. Silent regression happens when a model update, a skill edit, or even a context change makes things slowly worse without you noticing. And we all score things differently, so when someone says "this skill is good," that means absolutely nothing to the next person inheriting it.

A Tool That Actually Measures Something

I came across skilleval recently, and it's the first thing I've seen that takes this problem seriously. It's a CLI that runs A/B testing on your skills with actual rigor in under two minutes.

The approach is refreshingly honest: it does blind testing where your skill gets injected into one run while another runs without it, randomizes which output is "A" or "B" to kill position bias, and has a judge compare them. The scoring returns an actual margin—0 to 3 points—which gets converted into relative scores. No fluff. If your skill helped on half the tasks and hurt on the other half, the output tells you exactly that.

What I like most is the assertion system. Not every question needs LLM judgment. If your skill is supposed to generate a POST endpoint and it generates a GET instead, that's not subjective. You write assertions for the things that are objectively true, and the tool fails those tasks automatically.

Why This Actually Matters in Practice

I've been thinking about this wrong. I've treated SKILL.md files like they're either good or bad, when really they're just prompts, and prompts have tradeoffs. Your API design skill might improve output quality for REST endpoints while silencing useful caveats for async patterns. Without measurement, you never know the cost you're paying.

The multi-turn conversation support is clever too. Most real skills operate across multiple exchanges, not single isolated prompts. Testing the whole conversation chain catches interactions you'd miss in single-turn testing.

What really gets me is the regression detection. Every run gets saved to a history file. You can diff runs from different dates and see the exact moment things started getting worse. This is how you actually prevent "skill hell"—that place where you've iterated so much you can't remember what changed or why things broke.

Here's what I'd actually run:

# Initial baseline run
skilleval ./my-skill --tasks ./tasks.yaml --runs 3

# After making changes
skilleval ./my-skill --tasks ./tasks.yaml --runs 3

# See what changed
skilleval diff ./my-skill

# Or test skill versions head-to-head
skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml

The CI integration is table stakes these days. Gating PRs on skill effectiveness scores means you catch regressions before they ship. You set a threshold like "fail if effectiveness drops below 0.3" and the tool blocks the merge.

The Honest Conversation

Here's what I'd do differently: I'd lean harder into deterministic assertions early. They're cheap to run and catch obvious mistakes LLMs shouldn't be making anyway. Save the cross-model judging for the cases where judgment actually matters.

I also notice it uses Gemini Flash as the judge by default. That's pragmatic—Gemini Flash is cheap and surprisingly capable at comparison tasks. But I'd want to test whether the judge itself is consistent. If Gemini Flash contradicts itself 20% of the time, my confidence interval is useless.

The tool is honest about its gaps. Tool-call evaluation isn't there yet. Visual dashboards are missing. But v0.3.0 doing this much rigorously is further than I expected.

Next Move

I'm going to retrofit this into one of my existing projects next week. I want to see whether the skill I shipped last month was actually helping or just making me feel productive. Something tells me I won't like the answer.

If you're building with Claude and shipping skills without measurement, you're letting uncertainty compound. Try skilleval on something you've already shipped. Let me know what you find—I'm curious whether the skills we thought worked actually do.

Source: This post was inspired by "How to Know If Your Claude SKILL.md Actually Works" by Dev.to. Read the original article