Confirmation Hacking in AI Evaluation

Confirmation hacking is designing evaluations that confirm what you already believe. SigRank avoids it with content-free telemetry — measuring token counts, not curated content.

The best evaluation is one that could prove you wrong.

SigRank's content-free telemetry removes the main vector for confirmation hacking: selecting which content to evaluate.

What is confirmation hacking?

Confirmation hacking is the practice of designing evaluations that confirm what you already believe, rather than tests that could falsify your beliefs. In AI evaluation, this takes several forms:

Benchmark selection: choosing benchmarks where your model, output, or operator excels, while ignoring benchmarks where it performs poorly.
Prompt engineering: crafting test prompts that favor a particular system, rather than using standardized or adversarial prompts.
Output cherry-picking: selecting favorable outputs to evaluate while discarding unfavorable ones.
Selective reporting: publishing only the results that confirm your hypothesis, while burying the rest.

How SigRank avoids confirmation hacking

SigRank avoids confirmation hacking through three structural mechanisms:

1. Content-free telemetry. SigRank measures only token counts — the four pillars of input, output, cache-read, and cache-write. There is no content to cherry-pick. You cannot select favorable prompts, craft favorable outputs, or curate a favorable test set. The telemetry is numeric, objective, and complete.

2. Universal coverage. Yield (Υ) = (cache_read × output) / input² is computed from all real session telemetry, not a curated subset. Every session contributes. You cannot exclude bad sessions or include only good ones — the metric reflects the operator's actual performance across all their work.

3. Public results. Results are published on a public leaderboard. You cannot selectively report favorable outcomes — the full ranking is visible to everyone. This creates accountability and makes confirmation hacking structurally difficult.

Υ = (cache_read × output) / input²

The same principle as independent benchmarks

Independent model benchmarks like Vals AI exist because labs cannot be trusted to evaluate their own models — the incentive to confirmation-hack is too strong. Contamination-free benchmarks like LiveBench exist because training on benchmark data is a form of confirmation hacking. SigRank applies the same principle to the operator layer: independent, public, content-free evaluation that cannot be gamed through content selection.

The governance framework MO§ES defines the measurement specification, privacy boundaries, and public accountability requirements that make this possible. The evaluation is governed, auditable, and resistant to the biases that plague self-reported or content-based evaluations.

Public Operator Evals

The public evaluation layer for AI operators — ranked by Yield (Υ).

AI Evaluation

The four layers of AI evaluation and where the operator layer fits.

AI Evaluation Frameworks

From NIST AI RMF to SigRank — the landscape of evaluation frameworks.

Privacy-Preserving AI Telemetry

How content-free token telemetry enables evaluation without privacy risk.

FAQ

What is confirmation hacking in AI evaluation?: Confirmation hacking is the practice of designing evaluations that confirm what you already believe, rather than tests that could falsify your beliefs. In AI evaluation, this means choosing benchmarks, metrics, or test cases that favor a particular model, output, or operator — while ignoring cases where it performs poorly. It is a form of evaluation bias that produces misleading results.
How does SigRank avoid confirmation hacking?: SigRank avoids confirmation hacking through content-free telemetry. Because SigRank measures only token counts — input, output, cache-read, cache-write — and computes Yield (Υ) = (cache_read × output) / input², there is no opportunity to cherry-pick test cases, design favorable prompts, or select outputs that confirm a hypothesis. The metric is computed from all real session telemetry, not a curated subset. The evaluation is public, so results cannot be selectively reported.
Why is content-free telemetry important for avoiding bias?: Content-free telemetry eliminates the main vector for confirmation hacking: selecting which content to evaluate. When evaluations involve prompt content or output text, the evaluator can choose favorable examples. With token counts only, there is nothing to cherry-pick. The four token pillars (input, output, cache-read, cache-write) are numeric, objective, and complete — every session contributes, not just the ones that look good.
Can confirmation hacking happen in model benchmarks too?: Yes. Model benchmarks are vulnerable to confirmation hacking through benchmark selection, prompt engineering, and contamination. A lab can choose benchmarks where their model excels, craft prompts that favor their model, or train on benchmark data (contamination). This is why independent benchmarks like Vals AI and contamination-free benchmarks like LiveBench exist. SigRank applies the same principle to the operator layer: independent, public, content-free evaluation that cannot be gamed through content selection.

Loading rankings…

What is confirmation hacking?

Confirmation hacking is the practice of designing evaluations that confirm what you already believe, rather than tests that could falsify your beliefs. In AI evaluation, this takes several forms:

Benchmark selection: choosing benchmarks where your model, output, or operator excels, while ignoring benchmarks where it performs poorly.

Prompt engineering: crafting test prompts that favor a particular system, rather than using standardized or adversarial prompts.

Output cherry-picking: selecting favorable outputs to evaluate while discarding unfavorable ones.

Selective reporting: publishing only the results that confirm your hypothesis, while burying the rest.

How SigRank avoids confirmation hacking

SigRank avoids confirmation hacking through three structural mechanisms:

Υ = (cache_read × output) / input²

The same principle as independent benchmarks

FAQ

What is confirmation hacking in AI evaluation?

Confirmation hacking is the practice of designing evaluations that confirm what you already believe, rather than tests that could falsify your beliefs. In AI evaluation, this means choosing benchmarks, metrics, or test cases that favor a particular model, output, or operator — while ignoring cases where it performs poorly. It is a form of evaluation bias that produces misleading results.

How does SigRank avoid confirmation hacking?

SigRank avoids confirmation hacking through content-free telemetry. Because SigRank measures only token counts — input, output, cache-read, cache-write — and computes Yield (Υ) = (cache_read × output) / input², there is no opportunity to cherry-pick test cases, design favorable prompts, or select outputs that confirm a hypothesis. The metric is computed from all real session telemetry, not a curated subset. The evaluation is public, so results cannot be selectively reported.

Why is content-free telemetry important for avoiding bias?

Content-free telemetry eliminates the main vector for confirmation hacking: selecting which content to evaluate. When evaluations involve prompt content or output text, the evaluator can choose favorable examples. With token counts only, there is nothing to cherry-pick. The four token pillars (input, output, cache-read, cache-write) are numeric, objective, and complete — every session contributes, not just the ones that look good.

Can confirmation hacking happen in model benchmarks too?

Yes. Model benchmarks are vulnerable to confirmation hacking through benchmark selection, prompt engineering, and contamination. A lab can choose benchmarks where their model excels, craft prompts that favor their model, or train on benchmark data (contamination). This is why independent benchmarks like Vals AI and contamination-free benchmarks like LiveBench exist. SigRank applies the same principle to the operator layer: independent, public, content-free evaluation that cannot be gamed through content selection.