Evaluating AI

Evaluating AI without evaluating the operator is like evaluating a car without evaluating the driver. The operator layer is the missing piece — and SigRank fills it.

You evaluate the engine. You evaluate the transmission. You evaluate the destination. But who evaluates the driver?

The model is the engine. The agent is the transmission. The output is the destination. The operator is the driver. SigRank evaluates the driver.

The analogy

Evaluating AI without evaluating the operator is like evaluating a car without evaluating the driver. You can measure the engine's horsepower (model evals), the transmission's gear ratios (agent evals), and whether the car arrived at the right destination (output evals). But if you never measure the driver, you miss the most important variable.

Two drivers in the same car on the same track can post very different lap times. The difference is the driver's skill — how they brake, how they corner, how they manage momentum. In AI, two operators using the same model and the same tools can have 100× different Yield. The difference is the cascade architecture — how the human structures context reuse, output extraction, and input minimization.

How SigRank fills the gap

SigRank evaluates operators using Yield (Υ), a token-cascade efficiency score computed from real session telemetry.

Υ = (cache_read × output) / input²

Four token pillars: input, output, cache-read, cache-write. These are the only telemetry primitives needed.

Privacy-preserving: token counts only — never prompt content, never code, never conversation.

Public: results on a public leaderboard with class tiers from NOVICE to SINGULARITY.

Continuous: measured from real sessions, not one-time tests.

AI Evaluation

The four layers of AI evaluation and where the operator layer fits.

Public Operator Evals

The public evaluation layer for AI operators — ranked by Yield (Υ).

Model vs Agent vs Operator Evals

A detailed comparison of the three evaluation layers and why each matters.

How It Works

The full methodology behind Yield, token telemetry, and operator measurement.

FAQ

What does evaluating AI mean?: Evaluating AI means systematically assessing AI system performance. It includes model evaluation (benchmarks like MMLU, Vals AI), agent evaluation (task completion), output evaluation (quality scoring), and operator evaluation (how effectively a human uses AI). Evaluating AI is not one thing — it is a stack of evaluations, one per layer.
Why is the operator layer missing from AI evaluation?: Because measuring the human requires different telemetry than measuring the model. Model evals use standardized test prompts. Output evals use LLM-as-judge. But evaluating the operator requires token telemetry from real sessions — input, output, cache-read, cache-write — which no existing framework collected and analyzed publicly. SigRank is the first to do so.
How does SigRank fill the gap?: SigRank fills the operator layer gap with public, content-free, continuous evaluation. It measures operators using Yield (Υ) = (cache_read × output) / input², computed from four token pillars: input, output, cache-read, cache-write. Token counts only — never prompt content. The evaluation is public; the work is private. Operators are ranked on a public leaderboard with class tiers from NOVICE to SINGULARITY.
Is evaluating the operator really necessary?: Yes. Two operators using the same model, the same tools, and similar prompts can have 100× different Yield. The difference is the cascade architecture — how the human structures context reuse, output extraction, and input minimization. Model evals cannot see this. Agent evals cannot see this. Output evals cannot see this. Without operator evaluation, you are evaluating the car but not the driver.

Loading rankings…

The analogy

How SigRank fills the gap

SigRank evaluates operators using Yield (Υ), a token-cascade efficiency score computed from real session telemetry.

Υ = (cache_read × output) / input²

Four token pillars: input, output, cache-read, cache-write. These are the only telemetry primitives needed.

Privacy-preserving: token counts only — never prompt content, never code, never conversation.

Public: results on a public leaderboard with class tiers from NOVICE to SINGULARITY.

Continuous: measured from real sessions, not one-time tests.

FAQ

What does evaluating AI mean?

Evaluating AI means systematically assessing AI system performance. It includes model evaluation (benchmarks like MMLU, Vals AI), agent evaluation (task completion), output evaluation (quality scoring), and operator evaluation (how effectively a human uses AI). Evaluating AI is not one thing — it is a stack of evaluations, one per layer.

Why is the operator layer missing from AI evaluation?

Because measuring the human requires different telemetry than measuring the model. Model evals use standardized test prompts. Output evals use LLM-as-judge. But evaluating the operator requires token telemetry from real sessions — input, output, cache-read, cache-write — which no existing framework collected and analyzed publicly. SigRank is the first to do so.

How does SigRank fill the gap?

SigRank fills the operator layer gap with public, content-free, continuous evaluation. It measures operators using Yield (Υ) = (cache_read × output) / input², computed from four token pillars: input, output, cache-read, cache-write. Token counts only — never prompt content. The evaluation is public; the work is private. Operators are ranked on a public leaderboard with class tiers from NOVICE to SINGULARITY.

Is evaluating the operator really necessary?

Yes. Two operators using the same model, the same tools, and similar prompts can have 100× different Yield. The difference is the cascade architecture — how the human structures context reuse, output extraction, and input minimization. Model evals cannot see this. Agent evals cannot see this. Output evals cannot see this. Without operator evaluation, you are evaluating the car but not the driver.