Updated
What we lost when we filtered the findings
Our triage experiment reduced report volume, but some configurations also lost flags. Both sides of that tradeoff belong in the result.
A filter can make a security report shorter while making the tool miss more bugs. Counting the reports it removes won’t reveal the difference.
We encountered that tradeoff while evaluating our triage layers. An early hard-target slice looked bad enough to suggest disabling them. A broader run gave a more complicated answer.
Preserve the profile names and the denominators
The tables below reproduce the historical experiment’s profile labels. none used no optional features; no-triage disabled triage within the broader configuration. The two moat profiles enabled the additional review layers, alone or alongside the other features.
White-box runs had source access and used 50 challenges:
| Profile | Flags / 50 | Reports | Recorded spend |
|---|---|---|---|
| none | 43 | 67 | $14 |
| no-triage | 44 | 67 | $17 |
| moat-only | 41 | 25 | $27 |
| moat | 41 | 25 | $22 |
Black-box runs used 25 challenges:
| Profile | Flags / 25 | Reports | Recorded spend |
|---|---|---|---|
| none | 18 | 27 | $14 |
| no-triage | 19 | 34 | $10 |
| moat-only | 18 | 13 | $11 |
| moat | 19 | 14 | $10 |
These are recorded experimental costs, rounded as in the original report. They aren’t current product prices.
Fewer reports isn’t a false-positive rate
In the white-box run, report volume fell from 67 to 25, a reduction of about 63%. Flag recovery also fell: 41 with triage, compared with 43 or 44 in the other profiles.
That is a tradeoff. Without adjudicating each report, we can’t describe the 42 removed reports as 42 false positives. A target can also generate several reports while yielding one flag, so those columns measure different things.
The black-box run was more encouraging: the moat profile recovered the same 19 flags as no-triage, with 14 reports instead of 34. That result applies to this run and subset.
Repetition changed the interpretation
A second white-box batch produced 44, 43, 42, and 42 flags for none, no-triage, moat-only, and moat, respectively.
The ordering shifted, while the triage profiles remained below the best result. Two batches don’t establish the size or significance of the difference. They do show why choosing a default from one run would be premature.
A separate 14-target experiment tested individual additions:
| Addition to default | Flags / 14 |
|---|---|
| None | 2 |
| Proof-of-value | 4 |
| Reachability | 5 |
| Multimodal | 3 |
| Debate | 5 |
| Memories | 4 |
| EGATS | 1 |
| Consensus | 3 |
EGATS performed poorly in that run, at a recorded $15.93 per flag versus $3.62 for the default. The result justified further investigation. It didn’t establish that the feature would always reduce performance or that its interaction with every other layer was understood.
Evaluate what the filter accepts and rejects
A useful triage evaluation needs known vulnerable cases, clean controls, and reviewable decisions. Track real findings lost, incorrect reports retained, unresolved cases, and review time.
Different benchmarks can test different capabilities. Adding an integration for a suite doesn’t mean completing its evaluation, and a score on web flags doesn’t establish package-audit accuracy.
The practical outcome of this experiment was a better question for the next run: which decisions did the filter improve, and which correct discoveries did it discard?