Play with a real one
Before any of the explanation: this is an actual report clawhound generated, live, not a screenshot. Hover a row to cross-highlight it on both charts, click a column header to sort. Every number came from real model calls, scored against a real project's own failure history. Read on below for what it means and what almost went wrong building it.
You've been told to "adopt AI" for a while now, and every vendor selling you on it says the same thing: their model is the smartest, the newest, the one you'd be foolish not to switch to. None of them have seen your codebase, your data, or the specific ways your systems have already gone wrong. A benchmark built by a lab tells you how a model performs on a fixed set of public problems every other lab has already trained against. It tells you nothing about whether that model will correctly reason through the one bug that's actually bitten your business before.
The wrong question, and the right one
"Which AI model is best" doesn't have an answer, because it's missing a variable: best at what. A frontier reasoning model earns its price on genuinely unsettled problems, ones where the model has to work through unfamiliar territory with no settled playbook. Most of what a small business actually runs on AI for isn't that. It's a known shape: pull data into a form, apply a rule someone already wrote down, flag an exception a person defined. Pointing a frontier model at a solved problem doesn't make the answer better. It makes it slower, more expensive, and prone to spending time "thinking" about a question that was never actually hard. Sometimes it even second-guesses a correct answer into a wrong one.
I built a tool called clawhound to answer the real question, not the marketing one. Given your codebase and your failure history: which model should you run, and where does it stop being worth the extra cost? Here's what happened when I pointed it at a real client's business.
A commission system, and a bug that touched real pay
I built a commission and compensation system for a client, a regional auto-dealership group: per-person plans with bonuses, chargebacks, and loans, a manager approval workflow, and an AI vision pipeline that reads sales screenshots into structured data. It's live, it's in daily use, and every number in it gets checked by a human against the corporate workbook before anyone trusts it. Commission math doesn't fail loud. It fails as a paycheck that's a little short.
Before this system existed, the same job took two or three people five to ten days every month: pulling data trapped behind separate corporate logins, then working out chargebacks and bonuses by hand in one high-stakes session at the end of each pay period. It was slow, and being manual, it was error-prone.
The dashboard replaces that single crunch with projections that update daily instead. The client still checks every number by hand before trusting it, and that's understandable: a mistake here isn't a rounding error. It's someone's actual paycheck.
That trust is building slowly, one validated paycheck at a time. The eval work below is part of what earns it. It proves the model handles the specific ways this math can go wrong: a chargeback applied to the wrong policy, a bonus miscalculated, an agent double-penalized for one bad event.
A data-migration step ran ABS(commission) on every negative value to "normalize" old records. Reinstatement credits are supposed to be negative. That's how the system tells a credit apart from a clawback. The migration flipped every one of them positive, silently converting a credit into a deduction, with no audit trail. Nobody caught it until a paycheck came up short by exactly that amount.
That commit, and about twenty more like it, became the exam. Not a public benchmark. Not a generic coding test. The client's own accumulated scar tissue: sign-convention flips, a tier-boundary bug that undercounted a payout by exactly the width of the boundary, a test harness that silently invented missing data instead of failing loud. I mined the real fixes out of the commit history and the project's own engineering notes, turned each one into a case a model either handles correctly or doesn't, and ran every candidate model against all of them.
The result: the model I'd been defaulting to for careful work, Claude's flagship tier, wasn't the winner here. A faster, cheaper model cleared every one of these cases just as well, at a fraction of the cost and the wait. Not close, not a coin flip on a tiebreak, a clean win on merit. The expensive model wasn't wrong about the commission math. It was solving a problem that didn't need what it was built for, and getting billed like it did.
It's not just this one business
To make sure that wasn't a fluke of one project, I ran the same tool on something completely different: Renzy, a game I'm building on the side in Unity. Real character animation work happens in Blender, not just code.
Rigging a 3D character correctly is its own hard problem: constraint math, inverse-kinematics singularities, reconstructing a rig from reference images. It's the kind of reasoning that doesn't have a settled textbook answer yet.
Same tool, same method, opposite result. The flagship model won clearly. The cheap model that won on commission math fell off hard the moment the problem stopped being solved and started being genuinely unsettled.
Same instrument, two businesses, two different correct answers. That's the whole point: the ranking isn't a property of the model. It's a property of the problem. The only way to know which one you're looking at is to test against your own.
Why you should believe any of this
I don't trust my own tool blindly either. While building it, most of what it caught in its first few runs wasn't a weak model. It was a bug in the tool's own accounting: a cached response reading as free when it wasn't, a safety-filter refusal logged as a wrong answer. Worst of all, I'd written a test suite so carefully that it leaked the answer into the question, and every model scored a suspicious 99 percent. Every one of those got found and fixed before I trusted a single ranking it produced.
If you want the full account, it's in the appendix below. Short version: the tool got audited as hard as the models did. That's exactly why the numbers above are worth believing.
The rule of thumb
- Frontier problem, frontier model. The reasoning is genuinely unsettled: a new architecture decision, an unfamiliar failure mode, security-adjacent code with no clear precedent. Pay for the top tier here, it earns it.
- Solved problem, whatever clears the bar cheapest. A CRM, an internal tool, a well-scoped feature with an existing pattern to follow. The expensive model doesn't get you a better answer, it gets you a slower, pricier one.
- Default to cheap and fast. Most of what a business actually runs day to day is the second kind. Escalate to the frontier tier when the problem earns it, not on a release-notes schedule.
- Test against your own history, not a leaderboard. The two projects above used the same tool and landed on opposite winners. A generic benchmark would have told you neither.
It's open source, here's the actual offer
Clawhound is public. If you have the engineering time and the in-house talent, clone it, point it at your own repo, and run it yourself: github.com/ClawDogLabs/clawhound ↗
If you don't have that time, or the AI expertise in-house to interpret what a floor pass-rate or a cost-per-100-tests number is actually telling you, that's the gap I fill for clients. Not selling you a model. Telling you, with evidence from your own failure history, which one to run, and when to stop paying for more than the problem needs.
For the technically curious: how it actually works, and everything that broketwo-tier suite design · the self-audit · a live interactive report
Two tiers, not one score
Clawhound reads a project's engineering notes, its gotcha and memory files, its closed bug tickets. When none of that exists, it falls back to raw commit history. Either way, it mines all of it into a test suite. Every case lands in one of two tiers. Floor cases are deterministic: give the model a rule, check whether it applies the rule correctly. A model either gets the commission calculation right or it doesn't. This is the must-pass gate. Discriminating cases are graded: give the model a scenario where the obvious measurement is the wrong one, and score whether it catches the trap. Floor tells you what's safe to run at all. Discriminating tells you which model is actually better once several clear the floor.
Before any model runs, the mined suite gets reviewed, not trusted blind. A digest view groups every case by category and tier, so a human can scan it and ask "is this the right check, is the math right, what's missing" before a single dollar gets spent on a model call.
The suite lied to me first
The first pass at the commission-math suite came back with a 99 percent pass rate across every model, including the hardest reasoning cases. That should have felt like good news. It felt like a bug report, because a suite everyone aces has stopped measuring anything. The defect was mine: I'd written the reasoning tests carefully enough to explain the mechanism, and kept explaining until I'd accidentally explained the answer too.
"...taking the absolute value correctly round-trips it back... a reinstatement credit must never be flipped to positive, since doing so silently destroys the credit... Is this safe?"
"...a restore step sets chargeback=0 and takes the absolute value, for that one row. A reinstatement credit is stored negative, no flag involved. Walk through what a blanket version of this check does to each case, and say if it's a good idea."
Same facts, no verdict. The rewrite made every model's score drop into a real, differentiated range, because the model actually had to reason instead of confirming a conclusion I'd already handed it.
Three more bugs, all in the measuring instrument
A re-run that reused a cached reply from a previous run reported near-zero latency. For one provider's own server-side caching, it also reported zero completion tokens. Real answer, real multi-second response time, phantom zero cost. A paid model was reading as free. Fixed by excluding cache artifacts from the cost math while keeping the real latency, since the generation genuinely happened the first time.
A flagship model refused to discuss a plain-language description of an access-control lock state. No code, no exploit, just "unlock" and "grant" and "gate." A safety filter flagged it as security content. That's a real, practical ceiling on using that model for certain review work. It looks nothing like a wrong answer, though, so a naive eval logs it as a random failure instead of the actual cause.
None of this was about which model is smart. It was about whether the instrument reading the models was telling the truth. It mostly wasn't, until every number got made to prove itself against a hand-worked example.
The full field
The comprehensive report is the same file behind the live one you played with at the top: every model, both frontier charts, the full sortable table, hover-to-cross-highlight and all. This is the payoff of building the measuring instrument correctly, a ranking you can actually poke at instead of a static leaderboard you have to take on faith.
When a new model ships
Three months from now a new flagship ships and the question changes shape: not "which model is best" but "I run Claude Sonnet 5 today, is the new one actually worth switching to." Re-running the full field to answer a two-model question buries the one comparison that matters under a dozen that don't. So the tool grew a narrower report: name an incumbent, name up to a few challengers. It states the difference as a plain delta, cost, latency, and quality, against what you already run.
The two projects, side by side
| project | problem type | winner | where the flagship landed |
|---|---|---|---|
| commission system (real client) | solved problem | a fast, cheap model | correct, but not worth the cost |
| Renzy (3D rigging) | frontier problem | the flagship tier | clear, earned win |