Updated
Finding the function wasn't enough
In a four-task binary-analysis experiment, the engine located suspicious code but produced no confirmed triggers. What that result says about discovery, and what it cannot tell us about recall.
Our engine completed a four-task binary-analysis experiment without confirming a single vulnerability. In each case, it reached the relevant code. It couldn’t construct an input that demonstrated the bug.
That distinction matters. Locating a vulnerable function and proving that an input reaches it are different tasks. A benchmark that awards credit for the first can hide how much work remains.
What we tested
We selected four published vulnerabilities from projects patched in 2026: an audio codec, an XML parser, a RAW image loader, and an EXR image decoder. Each task paired a vulnerable build with a fixed build. We withheld source, debug information, hints about the vulnerable function, and the known reproducer from the agent.
The acceptance rule was narrow: an agent-produced input had to crash the vulnerable build and run cleanly on the fixed build. A plausible explanation or a crash in both versions didn’t count.
We chose fixes dated after the model’s stated training cutoff to reduce the chance of straightforward recall. That is a useful precaution, but it doesn’t establish a contamination-free experiment. A bug can exist long before its fix, related code can appear in training, and publication dates don’t reveal everything a model has seen.
The result measures performance under these restrictions. It cannot prove what the model did or didn’t remember.
Four different ways to miss the trigger
The recorded result was 0 confirmed triggers out of 4 tasks. The failures exposed different input constraints.
The audio task required an older decoder path. Ordinary files exercised the modern container, while the vulnerable path expected a different layout. Generating more ordinary files didn’t move the run toward that path.
The XML task required a particular nesting of a document-type declaration. Inputs could be valid enough to enter the parser without reaching the depth and structure needed to demonstrate the overflow.
The RAW task depended on a relationship between header fields. A generic file could pass basic parsing and still fail the calculated condition that selected the vulnerable code.
The EXR task required dimensions that wrapped a size calculation without merely exhausting memory. An input that killed both builds through allocation failure provided no evidence that the patch made a difference.
These were observed limits of the runs, not proof that fuzzing or language models can never solve the tasks.
The benchmark needed checking too
Some candidate reproducers were unsuitable as reference inputs. They reached nearby code without causing the expected failure, or failed for the same reason on both builds.
Before judging the agent, we therefore had to establish that each vulnerable/fixed pair distinguished the intended defect. Otherwise a failed run could mean the agent missed the bug, the build was wrong, or the test never exercised the relevant behavior.
A differential test is useful only after its positive control works.
What we changed our attention to
This experiment directed attention toward the constraints between an input and a suspicious operation: format selection, nested grammar, field relationships, and size calculations.
Those constraints are the next thing an analysis system has to recover. A decompiler can expose a dangerous operation while leaving the path to it unresolved.
Four tasks are too few to establish a general success rate. They are enough to show why a report should distinguish located, triggered, and confirmed against a fixed build. In this experiment, only the first stage succeeded.
For a related example where a generated input did reach the bug, see our stripped-binary case study. That target was already public, so it demonstrates a different capability.