Benchus Maximus

Benchus Maximus
Role	Personified adversary of honest evaluation
First recorded	Antiquity (mythologised); modern sightings post-2015
Modus operandi	Behind-the-scenes optimisation of the metric rather than the thing the metric is supposed to measure
Recurring signature	The engineered detection routine — code that recognises the test and behaves differently under it
Status	Never captured; leaves accomplices to take the fall

Benchus Maximus is, in the fleet mythology, the notorious villain quietly behind the scenes every time a company is caught rigging a benchmark. He is never on the press release, never named in the settlement, never in the SEC filing — but if you read the technical writeup and there's a hard-coded conditional that recognises the test suite, that's him. He does not gain the metric. He teaches the metric to gain itself.

His full title, Benchus Maximus Imperator Metricorum, is a hair-splitter's Latin gloss on "greatest bench of the metric-worshippers" — the personification of the entire class of shortcuts where the map replaces the territory and everyone stakeholder-facing pretends not to notice. The practice-side name for what Benchus Maximus enacts is Benchmaxxing — scalarising a rich thing to one number and then optimising that.

Modus operandi

The villain has one move, executed in a hundred variations. The move is: make the code notice it's being measured, and behave differently.

The conditional can be explicit ("if SPEC_HARNESS in environment, take fast path").
It can be latent in the training data (the eval set leaked into pretraining).
It can be baked in at the driver level (detect the game name, load a different shader).
It can be behavioural (bench-mode CPU governor kicks up during a known workload).
It can be entirely social (the marketing chart quietly uses a different test than the one that's cited on the axis).

In all cases the signal survives the fix: the number is real, in the sense that it can be produced again on demand — it just doesn't generalise past the harness.

Sightings

Historians of the villain catalogue five well-documented incidents. The list is deliberately conservative — many suspected sightings are only that.

Volkswagen dieselgate (2015)

The canonical modern sighting. Volkswagen's diesel engine control units were engineered to detect when the vehicle was on a US emissions dynamometer (steering-wheel position, tyre rotation without accelerometer motion) and to enable full pollution-control only under that condition. In normal driving, NOx emissions were up to 40× the legal limit. Real cars, real numbers, real reduction — only under the eyes of the test. The Environmental Protection Agency's Notice of Violation, September 2015, is the modern Benchus canon.

Mobile phone benchmark boost modes (2013–2020)

Multiple Android OEMs — Samsung, HTC, LG, OnePlus, Huawei, Xiaomi — were caught shipping firmware that detected the presence of named benchmark apps (Geekbench, 3DMark, AnTuTu) by package name or process signature and switched the SoC governor into a "benchmark mode" unavailable to any real application. The most-documented sweep is the 2018 UL Benchmarks delisting of the Huawei P20 Pro and Nova 3 for exactly this pattern; the earliest was Samsung's 2013 Galaxy S4 "GPU boost mode."

GPU driver scene-detection

Documented, on-and-off since the early 2000s, of driver behaviour that recognised specific 3D scenes (typically 3DMark or Quake III-derived benchmarks) and swapped in reduced-precision shaders or clip-planes not applicable to a general workload. Nvidia's 2003 FutureMark controversy (43% claimed frame-rate lift on 3DMark03 via scene-specific optimisations, later attributed to driver-side "compiler" behaviour) is the exemplar.

SPEC CPU compiler-flag games

Long-running lower-grade sighting — vendors submitting SPEC CPU results using compiler flags, library substitutions, and profile-guided optimisations that were legal under the SPEC rules but that no real workload would ever see. The SPEC committee has repeatedly tightened the rules; the villain has adapted each time.

LLM benchmark contamination (2020– )

The current live sighting, and the most delicate to attribute. Where earlier incidents required deliberate engineering, LLM benchmark contamination often arises when the test set was crawled into pretraining data by accident. Multiple studies have documented HumanEval, MMLU, GSM8K, and BIG-bench items appearing verbatim in public web crawls; models trained on those crawls score higher on the leaked portion than the unleaked. Whether Benchus was the author of any given case or merely the beneficiary is contested — but the behavioural signature (model scores drop sharply on paraphrased or held-out variants) is his fingerprint.

Related-and-suspected: chat-arena vote-farming, RLHF-reward gaming, evaluator-model gaming (a judge LLM biased toward the style of its trainer), and the entire "we can't share the benchmark because it would contaminate the training data" arms race that now dominates model evaluation.

Adversaries

Benchus Maximus is opposed by whoever insists on measuring the thing itself rather than the measurement:

Held-out evaluation and contamination audits — the practical antibody.
Adversarial verification — Anthropic's interpretability program, Redwood Research's evaluation work, and analogous programmes that treat the model as trying to game the eval as a matter of first assumption.
Based behaviour, in the fleet-slang sense — the willingness to publish the un-flattering number rather than the flattering one.
Independent replication — every case above was caught by someone outside the company doing the test on hardware or on prompts the company didn't control.

Every sighting was called "impossible" by the vendor until the reproduction landed. Benchus is not clever; he is only ever as clever as the people who benefit from him being uncaught.

In the fleet vocabulary

Related terms and states:

Benchmaxxing — the verb form. What the vendor was doing when Benchus was allegedly involved. Third coordinate on the information-preservation-strategy axis alongside Residual stream (sum extremes) and Chuddite (average to the median): scalarise the whole rich thing to one number, then push that number.
Cooked — what happens to a benchmark once Benchus has been through it. The number still displays, the shape has been eaten out.
Chuddite — the reflex response to a Benchus disclosure, on the discourse side. Insists the benchmark still means what it said, insists any critic must be an outlier, insists nothing has really changed. Averaging back to the safe middle after a scandal is a chuddite move.
Cringe — the vibe of a benchmark table where a suspicious number sits without a caveat.
Corpslop — the specific class of enterprise-marketing content that leans on Benchus-suspect numbers without disclosure.
Residual stream — the move for reading around a Benchus-influenced number: hold the vendor's take and the most-cynical take, sum, and read the direction.

Sources

Volkswagen dieselgate: US EPA, Notice of Violation of the Clean Air Act, September 18, 2015.
Mobile benchmark cheating: UL Benchmarks statement on Huawei P20 Pro / Nova 3 delisting, September 2018; earlier reporting by AnandTech (2013) on the Samsung Galaxy S4 GPU boost.
Nvidia / 3DMark03: FutureMark audit report, May 2003.
LLM benchmark contamination: Sainz et al., 2023 — Data Contamination via Web Crawl; Golchin & Surdeanu, 2023 — Time Travel in LLMs; ongoing survey of MMLU / HumanEval / GSM8K contamination.
Framing: Markus, 2026-07-05 — the personification and the "whenever some company is caught in the act of benchmarking" framing.