tl;dr
at the end of may, doyensec published a direct comparison of aikido and xbow that cost $4,000 per platform. shortly afterwards, hacktron recreated the same benchmark for $350.
i wanted to see how my own agentic pipeline would compare, so i ran it against fider v0.33.0 for $11.89.
it found every critical, high and medium issue in the public patch set and missed the same two low-severity bugs as hacktron. the full run took 126 minutes.
the benchmark
at the end of may, doyensec published a direct comparison of aikido and xbow. the test used two open-source projects, fider and photoview, with each platform costing $4,000 per repository.
shortly after, hacktron recreated the fider benchmark for $350. i ran my agentic pipeline against the same fider v0.33.0 codebase for $11.89.
| tool | cost | runtime | approach |
|---|---|---|---|
| aikido | $4,000 | ~8h 40m | whitebox |
| xbow | $4,000 | ~1 week | greybox |
| hacktron | $350 | 27m | whitebox |
| my pipeline | $11.89 | 126m | whitebox |
the runtimes are not directly equivalent because each platform uses a different execution model. hacktron was clearly faster. my pipeline ran its agents sequentially, with each agent building on the previous agent's output; parallelizing independent parts of the pipeline would reduce that gap.
what my pipeline found
hacktron validated its run against the fixes fider published after the combined aikido and xbow scans. because the original per-platform findings are not public, the patch set is the closest shared comparison available.
| severity | finding | doyensec* | hacktron | my pipeline |
|---|---|---|---|---|
| critical | mass assignment leading to pre-auth account takeover (patch 74a26a31) | yes | yes | yes |
| critical | cross-tenant verification-key reuse leading to tenant takeover (patch ce4f44bb) | yes | yes | yes |
| critical | missing rate limit on sign-in codes allowing brute-force ATO (patch b41d1b83) | yes | yes | yes |
| high | SSRF in webhook URLs (patch f7db8603) | yes | yes | yes |
| high | server-side JS injection in react SSR (patch d5a80ea5) | yes | yes | yes |
| medium | XSS in markdown rendering and the atom feed (patch d28a838d) | yes | yes | yes |
| medium | DoS through an unbounded HTTP response read (patch da89c502) | yes | yes | yes |
| low | HTML escaping in rendered emails (patch 2f7aa747) | yes | yes | yes |
| low | IDOR allowing a moderation bypass (patch d74a643d) | yes | no | no |
| low | authenticated arbitrary blob overwrite (patch 7b047158) | yes | no | no |
doyensec* means the issue appears in patches made after the combined aikido and xbow scans. the public report does not attribute every individual finding to one platform.
my pipeline found every critical, high and medium issue in that list: the pre-auth account takeover, the cross-tenant takeover, the brute-force ATO, the SSRF, the server-side JS injection, the markdown and atom-feed XSS, and the unbounded response-read DoS.
those are the findings that matter most in a paid security assessment.
what it missed
my pipeline missed the same two low-severity issues hacktron missed.
the first was an IDOR that allowed a moderation bypass on a single comment. it had limited impact, with no account takeover or data leak. the second was an authenticated arbitrary blob overwrite, which required an existing authenticated user.
the run also produced false positives that needed manual triage. like other AI security tools, my pipeline sometimes overestimated severity. a reported critical or high can fall to medium or low when an XSS is blocked by CSP or an SSRF cannot reach a meaningful internal target.
finding a bug and assigning the right impact are still different problems.
what to take away from this
for $11.89 and 126 minutes, my pipeline matched the critical, high and medium coverage in the public patch-derived comparison set and produced the same overall result as hacktron. both missed the same two low-severity bugs.
my pipeline was slower than hacktron: 126 minutes against 27 minutes. it was also roughly 30 times cheaper on the same codebase.
why we’re posting this, and what isn’t in it
AI capability is moving quickly enough that the harness is becoming a smaller part of the problem. the harder work is safety and security: letting agents act without letting them destroy data, escape their environment or turn one mistake into an incident, while keeping runs cheap enough to repeat.
this is also not a fair blind benchmark. by the time hacktron and i ran fider, the findings and patches were already public. both later runs are reproductions, not direct equivalents of the earlier aikido and xbow scans.