The model evals revolution
In 2023, AI model quality was marketing copy. Labs published cherry-picked benchmarks, selective comparisons, and curated examples. Then came public model evals \u2014 Vals AI, LMSYS Arena, LiveBench, Hugging Face Open LLM Leaderboard. These platforms ran standardized, public, reproducible evaluations. Overnight, model quality became measurable. You couldn't just claim your model was good \u2014 you had to show the scores.
The impact was immediate. Labs optimized for benchmarks. Users had real data for model selection. The leaderboard became the scoreboard. Public evals created accountability for AI labs the same way public financial reporting created accountability for companies.
The operator evals gap
Models are evaluated. Operators are not. There is no Vals AI for the human wielding the AI. No LMSYS Arena for the person writing the prompts. No LiveBench for the operator managing the context window. Operator skill is vibes \u2014 "they seem productive," "they're fast," "they ship a lot."
This is a problem because the skill gap is real. Two operators using the same model, the same tools, and the same prompts can have 100x different outcomes. The difference isn't the model \u2014 it's the cascade architecture. How efficiently does the operator reuse cached context? How lean is their fresh input? How much output do they extract per turn? These are measurable quantities. They're just not being measured publicly.
The thesis: operator skill should be public
Operator skill should be measurable, comparable, and public. Not because operators need to be ranked for ego \u2014 because accountability creates improvement. When model labs faced public evals, they got better. When operators face public evals, they get better. The leaderboard is the scoreboard.
The metric is Yield (\u03a5) \u2014 token-cascade efficiency. Yield measures whether signal is compounding or tokens are being burned. It's objective (measured from token counts, not opinions), privacy-preserving (no prompt content, just counts), and comparable (every operator gets the same metric from the same formula).
What public operator evals change
1. Skill becomes measurable
Before: "They're a good AI coder." After: "Their Yield is 2,462,656 \u2014 S-class, top 0.1%." Vibes become numbers.
2. Improvement becomes trackable
Operators can see their Yield trend over time. Are they getting better at context reuse? Is their input economy improving? The leaderboard shows weekly movement \u2014 who climbed, who dropped, who entered.
3. Hiring becomes data-driven
Companies hiring AI operators can look at public Yield scores instead of take-home assignments. The leaderboard is the portfolio. A S-class operator with 6 months of telemetry is a better signal than a 2-hour coding test.
4. Tooling becomes accountable
If operators using Claude Code consistently out-Yield operators using Cursor, that's public data. Tool selection becomes evidence-based, not preference-based.
The privacy contract
Public operator evals sound invasive. They're not. SigRank collects token counts only \u2014 input, output, cache_read, cache_write. Never prompt content. Never code. Never conversation. The evaluation is public; the work is private. Your Yield score is on the leaderboard; your code is not.
This is the same privacy contract as a fitness tracker. Your step count is public; your location is not. Your heart rate is public; your medical history is not. The metric is public; the activity is private.
The future: operator evals as infrastructure
Public model evals became infrastructure \u2014 every lab checks Vals AI before shipping. Public operator evals will become infrastructure too. Every operator will check their Yield before a performance review. Every team will track aggregate Yield as a productivity signal. Every hiring manager will look at the leaderboard before making an offer.
The thesis is simple: what gets measured publicly gets better publicly. Model evals proved it. Operator evals will prove it again.
Frequently asked questions
What is the public operator evals thesis?
The thesis: AI user skill should be measurable, comparable, and public. Public model evals (Vals AI, LMSYS Arena) created accountability for AI labs. Public operator evals create accountability for AI users — developers, coders, anyone using AI. What gets measured publicly gets better publicly.
Why should AI user skill be public?
Because accountability creates improvement. When model labs faced public benchmarks, they got better. When AI coders face public evals, they get better. The leaderboard is the scoreboard. Public evals turned model quality from marketing copy into measurable data; operator evals do the same for developer skill.
How do public benchmarks create accountability for AI developers?
Before public evals, AI user skill was vibes — 'they seem productive.' After public evals, it's Yield — a measurable, comparable, public score. Two developers using the same AI model can have 100x different Yield. Public benchmarks make that difference visible.
What is the privacy contract for observing AI users?
SigRank collects token counts only — input, output, cache_read, cache_write. Never prompt content. Never code. Never conversation. The score is public; the work is private. Same privacy contract as a fitness tracker: your step count is public, your location is not.
What score is used to benchmark AI coders?
Yield (Υ) = (cache_read × output) / input². It measures token-cascade efficiency — whether signal is compounding or tokens are being burned. Volume is noise; Yield is signal. Works across Claude, GPT, Gemini, Cursor, Copilot, and any AI coding agent.
How will operator evals become infrastructure?
Public model evals became infrastructure — every lab checks Vals AI before shipping. Public operator evals will become infrastructure too. Every developer will check their Yield before a performance review. Every team will track aggregate Yield. Every hiring manager will look at the leaderboard before making an offer.