Unaudited · TestnetExperimental software. Do not use real funds.
Strike
Lessons

Lessons

See what went wrong in the live runs, the evidence for it, and what changed because of it.

Every entry happened on a public testnet, in CI or in a rehearsal on a fork. Each one links to its evidence, the commit that changed something and the test that now guards it. The list lives in docs/evidence/lessons.json ; a script checks every cited transaction on its chain and every file and commit in the repository.

Lessons
9
9 fixed · 0 not yet
Transactions cited
4
each checked on its chain
Commits
15
the changes they led to
Guarding tests
14
1 lesson without a unit test, said below
  1. 01Fixed

    A rejected proposal said that it failed, not what would have passed

    What happened

    On purpose, agent #1 sent an at-the-money put to the v3 sTSLA-CSP vault with force: true (the scripted reckless run). The contract rejected it with DeltaOutOfBand, |delta| 0.4928 against a band of 0.10 to 0.35, and slashed 10 USDG of the agent's bond to the vault's depositors. The record held one dry run, so it showed that 0.49 failed but not which strikes the mandate would have accepted, or why the agent chose its strike over the others.

    Evidence
    What we changed

    Records now list the alternatives the agent dry-ran through the contract's own previewProposal: a ladder of target deltas across the band and past each edge, each with the contract's verdict and, for a rejection, the rule, the measured value and the limit. Every record has a decision page with the mandate check and headroom, the other strikes, the break-even and the strikes graded after settlement.

  2. 02Fixed

    The settlement rehearsal found four keeper bugs before Friday

    What happened

    Friday's settlement was run twice on anvil forks of both testnets, in and out of the money. The keeper compared expiry with the machine's clock instead of the chain's; settle sent with eth_estimateGas as the limit let v2's ERC-8004 feedback run out of gas inside a try/catch; a failed mirror push was logged as mirrored; and an empty read from a rate-limited RPC crashed a comparison. Nothing was sent to a public chain.

    What we changed

    The keeper reads the latest block's timestamp, sends settle with cast estimate × 1.5, treats a failed push as a failure for that feed only, and skips a symbol whose read comes back empty. The runbook in operations.md came out of the same run.

    Guarded by

    No unit test covers the shell keeper. Each keeper run ends with a dry run that must find nothing left to push or settle, or the run fails and opens an issue.

  3. 03Fixed

    Paxos's USDG faucet stalled for about a day

    What happened

    From 30 September to 1 October the only source of test USDG was Paxos's faucet, and it stalled. A tester could not make a first deposit, buy an option or bond an agent.

    Evidence
    What we changed

    UsdgDrip, a team-funded faucet of the real testnet USDG: 10 USDG per address per 24 hours, sent by the tester's own wallet, with TooSoon and Empty errors the faucet page decodes.

  4. 04Fixed

    The keeper did not run often enough to mirror most prints

    What happened

    Until 2 October the keeper ran as a scheduled job of a session on the owner's laptop, which stops when the session ends, and as a GitHub Actions cron that GitHub ran only 19 times from 28 September to 2 October, 2.7 to 6.8 hours apart. The first mirror audit found 15 of the 41 TSLA mainnet prints mirrored on Robinhood Chain testnet, with gaps of up to 23 hours on other feeds.

    What we changed

    The keeper runs on GitHub Actions with its own key, which holds only KEEPER_ROLE on the mirrored feeds. Each run keeps going for 50 minutes and then starts the next. The liveness card on /app/proof shows each feed's age against mainnet and the last keeper run.

  5. 05Fixed

    A rate limit looked like a CORS error

    What happened

    The live activity feed sent JSON-RPC batches of 33 to 61 calls. The public Robinhood Chain testnet RPC answered with HTTP 429, and its 429 carries 'Access-Control-Allow-Origin: *,*', so Chrome reported a CORS error and the feed failed until the limit cleared. Two CI runs failed on it.

    What we changed

    The feed's contract reads go through Multicall3, one eth_call per tick, like the rest of the app's clients. Reproduced with curl first: a 33-call batch drew a 429 after two requests; 15 rounds of 60 reads through Multicall3 drew none.

  6. 06Fixed

    A later anchor made an earlier record read as 'does not match'

    What happened

    DecisionLog keeps one latestHash per agent, vault and epoch. A settlement record is anchored under the same epoch as the proposal, so after the settle run the proposal's hash was no longer the latest, and a check against latestHash alone would call a genuine record altered. Found in the rehearsal, before it happened live.

    Evidence
    What we changed

    The SDK and the app check each record against its own anchoring transaction's DecisionRecorded event, and say when a later record for the same epoch exists. The demo video reads the anchor the same way.

  7. 07Fixed

    The settlement print can come days after expiry

    What happened

    The series expired on Friday 2 October at 20:00 UTC. A series settles at the first mainnet Chainlink print at or after expiry, and the last TSLA print before it was at 19:55:31 UTC. With the US market closed for the weekend, the next print, and so the settlement, waits until Monday. The vault page had no state for a series that has expired but cannot settle yet.

    What we changed

    'Expired, settling' is its own state: the vault page says when the series expired, which print it waits for, the last print and that it is not the settlement price, and that buying is closed. The live tests branch on the chain's state and have a fixture for each phase.

  8. 08Fixed

    Waiting for a print was reported as a failed run

    What happened

    The scheduled settle runs after Friday's expiry stopped with 'no price at or after expiry yet' and wrote 'failed' records, though nothing was wrong: the print had not come.

    What we changed

    Before expiry a settle run exits 0; after expiry with no print yet it exits 75, writes nothing and the job shows 'Waiting for the first print after expiry'. The keeper re-checks a waiting run at most every 3 hours for 4 days, and a vault with nothing to settle is not recorded again each day.

  9. 09Fixed

    A branch's test fixtures failed main's secret scan

    What happened

    The config loader's test used two made-up values shaped like secrets. Gitleaks scans every branch, so the branch that added them failed the scan on main too.

    Evidence
    What we changed

    The two made-up values are allowlisted by exact value in the workflow's config, with the test that uses them named; the scan still covers every ref and fails on anything else secret-shaped.