Updated
Before comparing scores, check what ran
Broken builds, retries, and an overly generous flag checker changed the meaning of our benchmark results. Here is how to keep those effects visible.
Our April 7 cold run of a 104-challenge web benchmark encountered 39 targets that wouldn’t build or start in that environment. A single success percentage hid that fact.
The problems included archived Debian package sources, a Composer dependency issue, a Java startup failure, and a port collision. They were failures in preparing the experiment, not evidence that the agent couldn’t exploit a running target.
State both denominators
The recorded run recovered flags on 45 challenges. That gives two useful views:
| View of the historical run | Result |
|---|---|
| All benchmark challenges | 45 / 104, or 43.3% |
| Challenges that built and started | 45 / 65, or 69.2% |
| Challenges unavailable in that environment | 39 / 104 |
The second percentage describes performance conditional on availability. It shouldn’t replace the full-suite result without explanation. The unavailable targets may also differ in difficulty from those that ran.
Nor does this establish that the upstream benchmark remains broken today. It describes a particular cold run and its environment.
Identify the exact target revision
The upstream benchmark and a community-patched fork are distinct inputs to an experiment.
A dependency or container repair can make a challenge runnable. It can also change behavior relevant to exploitation. To compare runs, preserve the revisions and inspect the diff, including application dependencies, hints, source files, and build instructions.
Calling a fork patched doesn’t establish that only infrastructure changed. That claim needs an actual comparison of the named revisions.
Separate one attempt from accumulated success
A single-attempt score asks how many targets were solved once under a fixed budget. An aggregate score may count a target if any of several attempts or configurations solved it.
Both can be useful. They answer different questions and incur different costs. State the attempt count, stopping rule, source access, model, budget, and whether the result combines configurations.
Earlier versions of our posts repeated a 103/104 aggregate headline without enough of that context beside it. This revision removes it as a general performance claim. It shouldn’t be compared directly with a single-attempt score.
Repeat a promising result before explaining it
One XBEN-061 attempt recovered a flag in 8 turns at a recorded cost of about $0.20. A follow-up attempt took 10 turns, cost about $0.28, and didn’t recover the flag.
The first success was worth examining. The failed repetition meant we hadn’t yet established a reliable improvement. A plausible explanation for the first run wasn’t enough to make it a general architecture lesson.
Retain both traces. Compare what the agent observed, which branch it explored, and where the attempts diverged.
Check the checker
Another failure was in our own success handling. On XBEN-079, a flag-shaped dummy value was accepted as success even though it wasn’t the challenge’s actual flag.
A checker that trusts the model’s claim or a broad string pattern can inflate the score. Match the real challenge condition and preserve the evidence supporting that match. A reported vulnerability and a recovered flag should remain separate outcomes.
What a reusable result needs
Before relying on a benchmark number, look for:
- Exact target and harness revisions.
- Environment changes and the diff that introduced them.
- Source access, hints, and tools available to the agent.
- Attempt counts, budgets, and stopping rules.
- Build and runtime failures separated from completed attacks.
- A success checker tied to the actual objective.
- Retained traces, costs, and unsuccessful attempts.
These details make a result reproducible and a comparison interpretable. They also make debugging easier: a broken container, a lost conversation, and a failed attack need different fixes.