Lessons
See what went wrong in the live runs, the evidence for it, and what changed because of it.
Every entry happened on a public testnet, in CI or in a rehearsal on a fork. Each one links to its evidence, the commit that changed something and the test that now guards it. The list lives in docs/evidence/lessons.json ; a script checks every cited transaction on its chain and every file and commit in the repository.
- 01Fixed
A rejected proposal said that it failed, not what would have passed
What happenedOn purpose, agent #1 sent an at-the-money put to the v3 sTSLA-CSP vault with force: true (the scripted reckless run). The contract rejected it with DeltaOutOfBand, |delta| 0.4928 against a band of 0.10 to 0.35, and slashed 10 USDG of the agent's bond to the vault's depositors. The record held one dry run, so it showed that 0.49 failed but not which strikes the mandate would have accepted, or why the agent chose its strike over the others.
Evidence- proposeSeries, rejected and slashed (Robinhood Chain testnet, block 127183195)
- The decision record
What we changedRecords now list the alternatives the agent dry-ran through the contract's own previewProposal: a ladder of target deltas across the band and past each edge, each with the contract's verdict and, for a rejection, the rule, the measured value and the limit. Every record has a decision page with the mandate check and headroom, the other strikes, the break-even and the strikes graded after settlement.
- 02Fixed
The settlement rehearsal found four keeper bugs before Friday
What happenedFriday's settlement was run twice on anvil forks of both testnets, in and out of the money. The keeper compared expiry with the machine's clock instead of the chain's; settle sent with eth_estimateGas as the limit let v2's ERC-8004 feedback run out of gas inside a try/catch; a failed mirror push was logged as mirrored; and an empty read from a rate-limited RPC crashed a comparison. Nothing was sent to a public chain.
EvidenceWhat we changedThe keeper reads the latest block's timestamp, sends settle with cast estimate × 1.5, treats a failed push as a failure for that feed only, and skips a symbol whose read comes back empty. The runbook in operations.md came out of the same run.
Guarded byNo unit test covers the shell keeper. Each keeper run ends with a dry run that must find nothing left to push or settle, or the run fails and opens an issue.
- 03Fixed
Paxos's USDG faucet stalled for about a day
What happenedFrom 30 September to 1 October the only source of test USDG was Paxos's faucet, and it stalled. A tester could not make a first deposit, buy an option or bond an agent.
Evidence- Decision D38
- UsdgDrip deployed (Robinhood Chain testnet, block 127311607)
What we changedUsdgDrip, a team-funded faucet of the real testnet USDG: 10 USDG per address per 24 hours, sent by the tester's own wallet, with TooSoon and Empty errors the faucet page decodes.
- 04Fixed
The keeper did not run often enough to mirror most prints
What happenedUntil 2 October the keeper ran as a scheduled job of a session on the owner's laptop, which stops when the session ends, and as a GitHub Actions cron that GitHub ran only 19 times from 28 September to 2 October, 2.7 to 6.8 hours apart. The first mirror audit found 15 of the 41 TSLA mainnet prints mirrored on Robinhood Chain testnet, with gaps of up to 23 hours on other feeds.
What we changedThe keeper runs on GitHub Actions with its own key, which holds only KEEPER_ROLE on the mirrored feeds. Each run keeps going for 50 minutes and then starts the next. The liveness card on /app/proof shows each feed's age against mainnet and the last keeper run.
Guarded by - 05Fixed
A rate limit looked like a CORS error
What happenedThe live activity feed sent JSON-RPC batches of 33 to 61 calls. The public Robinhood Chain testnet RPC answered with HTTP 429, and its 429 carries 'Access-Control-Allow-Origin: *,*', so Chrome reported a CORS error and the feed failed until the limit cleared. Two CI runs failed on it.
EvidenceWhat we changedThe feed's contract reads go through Multicall3, one eth_call per tick, like the rest of the app's clients. Reproduced with curl first: a 33-call batch drew a 429 after two requests; 15 rounds of 60 reads through Multicall3 drew none.
Guarded by - 06Fixed
A later anchor made an earlier record read as 'does not match'
What happenedDecisionLog keeps one latestHash per agent, vault and epoch. A settlement record is anchored under the same epoch as the proposal, so after the settle run the proposal's hash was no longer the latest, and a check against latestHash alone would call a genuine record altered. Found in the rehearsal, before it happened live.
Evidence- Proposal anchored, sTSLA-CC epoch 1 (Robinhood Chain testnet, block 127182818)
- Settle record anchored, same vault and epoch (Robinhood Chain testnet, block 127834965)
What we changedThe SDK and the app check each record against its own anchoring transaction's DecisionRecorded event, and say when a later record for the same epoch exists. The demo video reads the anchor the same way.
- 07Fixed
The settlement print can come days after expiry
What happenedThe series expired on Friday 2 October at 20:00 UTC. A series settles at the first mainnet Chainlink print at or after expiry, and the last TSLA print before it was at 19:55:31 UTC. With the US market closed for the weekend, the next print, and so the settlement, waits until Monday. The vault page had no state for a series that has expired but cannot settle yet.
EvidenceWhat we changed'Expired, settling' is its own state: the vault page says when the series expired, which print it waits for, the last print and that it is not the settlement price, and that buying is closed. The live tests branch on the chain's state and have a fixture for each phase.
Guarded by - 08Fixed
Waiting for a print was reported as a failed run
What happenedThe scheduled settle runs after Friday's expiry stopped with 'no price at or after expiry yet' and wrote 'failed' records, though nothing was wrong: the print had not come.
What we changedBefore expiry a settle run exits 0; after expiry with no print yet it exits 75, writes nothing and the job shows 'Waiting for the first print after expiry'. The keeper re-checks a waiting run at most every 3 hours for 4 days, and a vault with nothing to settle is not recorded again each day.
Guarded by - 09Fixed
A branch's test fixtures failed main's secret scan
What happenedThe config loader's test used two made-up values shaped like secrets. Gitleaks scans every branch, so the branch that added them failed the scan on main too.
EvidenceWhat we changedThe two made-up values are allowlisted by exact value in the workflow's config, with the test that uses them named; the scan still covers every ref and fails on anything else secret-shaped.
Guarded by