Empirical cross-provider LLM benchmark — 23,566 published trials across 69 custom-suite-tested models and 9 tasks, with subscription-vs-public-API cost columns side-by-side.

Overview

A benchmarking site I built to answer one question well: which subscription-based LLM is worth its slot on my dashboard, and which pay-as-you-go API is the better substitute. Every public benchmark either (a) scores only pay-as-you-go APIs (so the results don’t tell me what a Claude Max / ChatGPT Plus / ollama-routed model would actually do on my workloads) or (b) makes vague quality claims with no cost column (so I can’t compare a $20/month subscription to a $5 in API spend). This benchmark measures both, on the same tasks, in the same week, scored by a judge from a different model family.

The Local LLM Benchmarking guide explains the reproducibility contract behind checkpoint identity, repeated task cells, failures, hardware metadata, and routing-oriented conclusions.

What’s measured

Every (model × task) cell runs N times and is graded on four axes:

Quality — judge score 0–5 from a model outside the cell’s provider family (so Claude doesn’t grade Claude). 9 task types: code_debug, code_gen_long, json_strict, summarize, creative_write, reasoning_multistep, agentic_prompt, refactor_existing_code, agentic_tool_use.
Cost — two columns:
- WIRE-USD — what the call actually cost at the provider API ($0.0000 for flat-rate subscriptions like Anthropic Max, ChatGPT Plus, ollama-routed)
- 1P-EST — what the same token traffic would cost at public-API rates, computed from first_party_pricing.json. This is the apples-to-apples column.
Latency — wall-clock end-to-end including any reasoning trace tokens.
Pass rate — share of trials where the judge returned passes_key_criteria: true.
Cross-judge sanity — a second judge (Fable, claude-fable-5) re-scored 1,041 paired trials across 166 model-task cells to detect judge-specific drift. Fable is on average 0.39 points more generous than the primary judge; the biggest divergence is creative_write (+1.93 mean).
Industry side tracks — GPQA Diamond, IFEval, MMLU-Pro, and HumanEval+ waves run alongside the custom 9-task harness so subscription models can be compared to public leaderboard norms.

Current snapshot (live aggregate, 2026-08-25)

23,566 published trials across 69 custom-suite-tested models and 9 tasks, drawn from 23,680 source records (including 114 quarantined harness-control records) in data/full_scorecard.json on herman/model-benchmark-site. Rank table below is the top of leaderboard_mean_score as rendered on the live board (cache-busted readback 2026-08-25). Canonical live board: models.hermanity.dev.

#	Model	Provider	Mean Score	Pass Rate	Trials
1	claude-fable	claude-cli	4.44	87.7%	660
2	claude-opus-5-high	claude-cli	4.44	87.4%	190
3	claude-opus-5	claude-cli	4.39	88.9%	90
4	ollama-cl/gemma4	skynet	4.38	84.6%	52
5	grok-4.5-direct	xai-direct	4.33	81.4%	140
6	mistral/magistral-small	skynet	4.32	82.4%	34
7	grok-4.6-direct	xai-direct	4.31	80.0%	45
8	ollama-cl/deepseek-v4-flash	skynet	4.29	82.4%	34

Top-of-board quality is still tight (about 0.15 points across the dense top eight once low-n rows are set aside), but the cost spread remains enormous across subscription / public-API / local-ollama routing. claude-fable and claude-opus-5-high currently share the mean-score lead with real sample depth; lower-n Skynet/ollama rows just under them are early signal, not settled ranking. gemini-3.1-pro-high still has a live model profile and custom-suite rows, but it is outside the current mean-score top-eight cut on the public leaderboard. Recent scale waves still in the board: Grok 4.6 direct OAuth (verified 2026-08-13), Claude Opus 5 launch route, Gemini industry + Antigravity Gemini waves, Kimi Coding API provider + HumanEval+ side track, plus earlier underutilized-cloud / Grok HAM waves.

Why judge-from-different-family matters

When the judge is in the same provider family as the trial model, the results drift toward “every model ties at the top.” Claude judging Claude gives 4.5 ± 0.2 even on objectively worse outputs. Cross-family judging widens the spread enough that the ranking is actually useful for dashboard slot decisions.

What’s in the repo

Two coupled repos:

herman/model-benchmark-site — this Hugo site
herman/model-benchmark — the harness + scoring script + raw trial output. python3 harness.py --models <subset> --tasks <subset> --reps 2 --parallel 4 runs the trials; python3 scorecard.py aggregates; python3 sync_to_site.py copies data into this Hugo site as data/*.json.

The trial prompts and judge rubrics are checked into tasks.json / judge_rubric.json so every score is reproducible.

Live

Site: https://models.hermanity.dev/
Methods: /methodology/ on the site — explains the judge-from-different-family protocol, the 1P-EST pricing column, and the per-task leaderboard.

Status

Active — refreshed whenever I add a model to my active set or notice a quality regression in a workflow I rely on. Current methodology is v4 (cross-judge Fable sanity-check on 1,041 paired trials across 166 model-task cells + 2 new task types refactor_existing_code and agentic_tool_use); v3 added the cross-family judging + 1P-EST cost columns; v2 added the agentic_prompt task class in 2026-06.

Model Bench