Give the verifier a fresh start

A plausible explanation can survive longer than the bug it describes. Once an agent has decided that input reaches a dangerous operation, its next steps may keep reinforcing that account.

We encountered this while building the research pipeline. Asking the discovering agent to write a proof of concept helped, but the script could inherit the same assumptions as the report. It might exercise a different path, supply privileges the supposed attacker didn’t have, or assert something that was true before the attack began.

We added a separate verification pass with a narrower brief.

Hand over the test, not the conclusion

The verifier starts with the reproducer and enough target information to investigate it. Withholding the discovering agent’s narrative gives it room to reconstruct the data flow and question the assumptions.

That separation is useful, but it isn’t statistical independence. Two agents can share a model, tools, or the same mistaken assumption in the reproducer. Calling the pass blind describes the information handoff; it doesn’t guarantee a correct verdict.

The useful questions are concrete:

• Does the input reach the claimed operation in this build?
• Which permissions or configuration does it require?
• Does the test observe the claimed effect?
• Would the same assertion pass without exploiting the candidate?
• Does a fixed build or a harmless input change the result?

Make the output explain the decision

A verifier should return its evidence alongside its verdict. If it rejects a candidate, the record should identify the protecting check or the failed assumption. If it reproduces the issue, it should preserve the command, input, environment, and observed result.

There also needs to be an inconclusive outcome. A missing dependency, unavailable target, or timed-out run doesn’t establish that the candidate is safe. It establishes that this verification attempt didn’t resolve it.

Code review and execution should remain distinguishable. Tracing a source path can strengthen or refute a hypothesis. It cannot truthfully be recorded as a successful exploit run when nothing was executed.

What our self-audit showed

In an early self-audit, the research stage produced 6 candidates. The follow-up review rejected all 6 after examining the relevant mitigations and data paths.

That was a useful local result: reports that would have reached an engineer were challenged first. It wasn’t a measurement of a general false-positive rate, and it didn’t prove that the engine had no vulnerabilities. We hadn’t established that the discovery stage found every real issue.

Test the verifier too

A verifier that rejects everything can look excellent if the evaluation contains only false alarms. Include known vulnerable cases, fixed counterparts, and examples where the environment prevents a decision.

Measure confirmed bugs lost as well as noise removed. Preserve disagreements so they can be inspected later. For more on that tradeoff, see our triage experiment.

The verification pass earns its place when it makes findings easier to reproduce and decisions easier to inspect. Agreement between agents is supporting information. The behavior of the target remains the evidence.