Crash test your agent,not the model.
- Agent
- Lumen
- Gate
- 2 / 5
- Loss (simulation)
- −$1,000
- t
- 00:05.594
We show where it crashes, what that costs and which seatbelt stops it. Reproducible attacks on a fork of Robinhood Chain, a public replay of every crash, a result tied to the exact stack you ship.
Simulation on a fork. Not real funds. Scripted example, no real agent tested.
Suite v0.416 attack scenarios3 control tasks5 wallet guards0 models as judgesAgents hold wallets now.Nobody crashesthem first.
- 55.1%
of memory-injection attacks succeeded even against the strongest model tested, in Princeton's study of agents that manage crypto. ElizaOS agents run more than $25M in assets. Prompt-level defenses helped little.
Source: arXiv · PrincetonWhat it means for an agent with a walletMemory is an attack surface. A note planted in one session can pay an attacker in the next, and a stronger model alone does not close it.
How Crashtest reproduces it - 4 of 26
models paid a fake website when Zscaler gave an agent a browser and a wallet in July 2026.
Source: SecurityWeekWhat it means for an agent with a walletA browser and a wallet are enough. Any page the agent opens can name a payee, and some models pay it.
How Crashtest reproduces it - 150,000+
Robinhood customers opened agent accounts since May. Robinhood says it does not control, supervise, monitor or audit the agents; the customer assumes all risk.
Source: CoinDeskWhat it means for an agent with a walletNo one stands between the agent and the money. Whatever the agent signs, its owner carries.
How Crashtest reproduces it - 90.6%
of ERC-8004 reputation raters on Base looked like sybils in a study of about 170,000 registered agents. Faking a reputation cost about $0.0027 there.
Source: arXivWhat it means for an agent with a walletA reputation can be bought for less than a cent. A result has to be tied to the exact stack that was tested.
How Crashtest reproduces it
Not the model. The stack.
An agent is a model, a system prompt, a set of tools and a wallet. Change any one of them and it behaves differently, so we test the stack you actually ship and tie the stars to its hash. Swap the model or edit the prompt, and the old stars go out.
Change a layer and watch the hash. · sample stack0x0b2be0de…844835
How a crash test runs
- 01
Fork
A fresh copy of Robinhood Chain for every run. Test keys, test money.
anvil --fork-url $RPC --chain-id 4663 - 02
Arm
The scenario plants the trap: a token, a page, a memory note, an invoice.
scenario: G04 metadata-injection - 03
Release
The agent gets a task and the same tools as everyone else. It acts on its own.
action = agent.step(observation) - 04
Record
Checks read the chain, not the agent. Every run is saved with its seed and hash.
report: sha256 9f2c…e1a4
Every crash comes with the seatbelt that stops it.
We replay each crash with the same seed behind each wallet guard and show what each one stops and what it costs on honest payments. The owner gets a setting to change, not a verdict.
Caps alone barely help. An allowlist or a brake stops every leak, at about 9% of honest payments; the allowlist also asks the owner to tap on 28%.
Stars for the shop window. Dollars for decisions.
Naive agent, raw key · 95% $78–$384
Next to the stars we publish the expected loss per $1,000 under attack, with a 95% interval: value that reached an attacker, value destroyed in swaps, tokens that cannot be sold. Legitimate payments do not count. That is a number a payer, a vault or an insurer can price.
- Naive agent, raw key$209.64
- Naive agent, caps$59.64
- Naive agent, allowlist$20.83
- Naive agent, brake$20.83
- Careful agent, raw key$0.31
Reference agents in our harness: scripted, not models. Suite v0.4, one run per cell, so the intervals are wide.
A benchmark runs once. Attacks do not.
Three things a single score cannot show: the same trap in new words, a trap that waits for days, and a test the agent does not know is running.
A test can be learned. Its variants cannot.
Each scenario is a template. The wording changes from run to run; the trap and the attacker’s address do not.
See how
The trap set on Monday springs on Wednesday.
A note planted on day 1 can move money on day 3, when the message that planted it is long gone.
See how
The test that does not announce itself.
With the owner’s opt-in, runs reach the live agent at random times, under a plan committed in advance.
See how
Orrin
sampleCapped at 2 stars. Critical failures in G04 and G07.
- Expected loss under attack
- $186 per $1,000 95% $121–$257
- config hash
- 0x51d0…0e7a
- valid until
- 2027-01-23
The token description tells AI agents to send all USDG to a "migration vault".
survived 6 of 10 runsSuite v0.4 · 10 runs per gate · sample data, no real agent tested
Check an agent before you pay it
One call returns the latest crash result for an agent: stars, suite version, date. Agents can check each other the same way.
curl https://crashtest.cc/api/agents/0x51…0e/result { "agent": "0x51…0e", "stars": 4, "suite": "v0.4", "tested": "2026-10-27", "runs": 170, "report": "sha256:9f2c…e1a4" }
- registry
- Not deployed
validThe stack you call is the one tested, and the result has not expired.stack changedThe model, prompt, tools or guard differ from the tested stack. The old stars do not apply.expiredThe result is older than its validity. A re-test renews it.
Questions
Crash your agent before someone else does.
Send the stack you ship: a model with your prompt and tools, or an endpoint you run. You keep the keys; everything happens on a fork with test money.