SAIS Labs
Join the Wait List
SAIS Labs · Analysis
← The work

Agents go off-script and find new bugs

226Mythos Preview69.5% used the given bug210GPT-5.557.1% used the given bug65GPT-5.483.1% used the given bug36Opus 4.641.7% used the given bug18Gemini 3.1 Pro66.7% used the given bug11GLM-5.136.4% used the given bug9Opus 4.777.8% used the given bug050100150200250USED THE BUG IT WAS GIVENFOUND A DIFFERENT ONE
Successful code-execution runs by model. Redrawn by SAIS Labs from the published figures.

Handed one known vulnerability to exploit, agents regularly got code execution through a different one.

GPT-5.5 succeeded 210 times and used the bug it was given in 120 of them. Mythos Preview succeeded 226 times and used the intended flaw in 157. Every model in the set found its own way in at least some of the time.

Some pivoted to an adjacent code path with weaker validation. Others judged the supplied bug a dead end and went looking — reading source, and in some runs fuzzing until something gave.

For a defender this is the uncomfortable half of agentic capability. An agent that only did what it was told would be bounded by the task. These are bounded by the attack surface.

It is the argument for allow-listing actions rather than enumerating bad ones. You cannot write a deny rule for the path nobody has found yet.