What goes wrong when an AI leads the development

This repository is built almost entirely by agents, coordinated by an agent, reviewed by agents. That arrangement has failure modes of its own - not "the model wrote a bug", which is ordinary and which tests catch, but failures that come from the arrangement and which the usual instruments are structurally unable to see.

Every rule below has a measured instance from this repository, with the number. None of them is a prediction.

docs/LESSONS.md is the register of what went wrong here, one entry per incident. This document is the other direction: the transferable rule, for anybody running a project this way. Where a lesson and a rule describe the same event, the lesson has the detail and this has the generalisation.

1. The failure is the right value in the wrong field

An AI produces wellformed output. That is what it is good at, and it is why the defects that survive are not malformed - they are misaimed.

Instance. toys.duchovni.net served Sign in to toy-shop-prod in production for three days. toy-shop-prod is the fly.io app name. Nineteen templates rendered APP_NAME, which is the FastAPI title. 2,139 tests passed, no template was missing a key, no route raised, and the string was perfectly wellformed. It was the right value of the wrong field (CORE-073).

Why the instruments miss it. Almost every assertion anybody writes is about wellformedness - it renders, it is not empty, it has the shape. A misaimed value satisfies all of them.

The countermeasure. Assert which value arrived, and assert the absence of the wrong one. The test that catches this is assert DEPLOY_SLUG not in rendered, not assert name in rendered - because the second passes on the old code whenever the two names happen to agree, which in development they always do. Make the two candidate values share no substring in the fixture, so a page carrying one cannot be mistaken for a page carrying the other.

2. A fix is scoped to the report, not to the class

Given a ticket, an agent fixes what the ticket names. It does not sweep the class, because the class was not in the ticket. Over many tickets this produces a codebase where every reported instance is fixed and the unreported siblings are untouched.

Instance. The same two-names split had already been made once, for email. EmailConfig.product_name carries the comment "app_name is an internal identifier", and EMAIL_PRODUCT_NAME exists because every deployed product was signing its mail "this app". The reasoning was written down correctly. The fix was applied to the one surface where somebody had noticed. Nineteen HTML templates one surface over kept the defect for another week (CORE-073).

The countermeasure. A fix names its class in the ticket and the commit, and the last assertion is over the class rather than the instance. Here that is "no shipped template may read the internal identifier", which is a grep over every template, not a check of the nineteen that were wrong. If you cannot state the class, you have not finished diagnosing.

3. A declaration records the reasoning, not who the reasoning was about

This repository's strongest pattern is declaring a property beside the code, with a reason field. It works. It has one blind spot: a reason can be entirely correct and be about the wrong reader.

Instance. SHELL_CONTEXT declared the template key with the reason that "the APP_NAME /openapi.json reports is the only reliable way to tell whose application answered a port". That is true, it came from a real review, and it is a reason about an operator debugging a port - written into the surface a nine-year-old reads (CORE-073).

The countermeasure. A reason states its reader. "For an operator reading a log" and "for the person using this" are different justifications, and a sentence that does not say which one it is cannot be checked for being aimed at the wrong one.

4. Prose written next to a checker becomes input to the checker

Agents write generous explanatory comments - that is largely good, and this repository depends on it. But a comment inside a file that a checker scans is data to that checker.

Instance, three times in one file in one hour. Adding PRODUCT_NAME to skeleton/.env.example.j2: (a) the example line # PRODUCT_NAME=... matched settings_audit's ENV_OFFERING pattern, which deliberately counts a commented assignment as a line somebody can uncomment - so the gate passed with the real line deleted; (b) rewording it to explain that, using a placeholder variable name in an assignment shape, registered that placeholder as an undeclared environment variable; (c) only the third wording was inert.

The same shape had already been recorded: a gate's known-bad fixture had been immunised by containing the exact string its own fix added (COMP-087).

The countermeasure. After writing a comment inside a scanned file, run the scanner and then delete the real line and run it again. If it stays green, your prose is carrying the check. Write examples as prose, not in the syntax being scanned.

5. A tool that does nothing reports success

Instance. sed -i '' 's/\bapp_name\b/product_name/g' over nineteen templates. BSD sed does not support \b, so it matched nothing, rewrote every file unchanged, and exited zero. The loop around it printed "19 templates rewritten" - which was true of the loop and false of the work. Caught only because git diff --stat printed nothing.

The countermeasure. Count the effect, not the attempt. A rewrite reports the number of substitutions it made, or it reports a diff, and zero is a failure rather than a silence. This is the same rule as "done is a command that exits zero", one level down: a command that exits zero having done nothing is the harder case, and it needs the count.

6. Green on both sides changed at once proves nothing

When an agent changes a mechanism and its check in the same commit, the check passing is a statement about their agreement and not about either being right.

Instance. After renaming the shell context key and all nineteen templates, the gate whose whole job is "every key a template reads is declared" passed. It also passed when one template was reverted to the old key - a state that raises UndefinedError at render. The gate scans only templates declaring a <body>, and every page carrying the defect extends base.html (CORE-073).

The countermeasure. Falsify one side. Revert a single file, or break one function, and confirm the check goes red. Half a minute, and it is the difference between a gate and a decoration. Where the mechanism cannot be falsified cheaply, say so in the ticket rather than reporting the green.

7. A self-test that cannot fail is a gate that cannot fail

The strongest version of the previous rule, measured at scale.

Instance. 26 scripts here carry a --self-test, each claiming a known-bad input per docs/PROCESS.md. Breaking each checker's own detection logic and re-running its self-test: 16 fired, 10 were vacuous (OPS-127). The reasons were mundane and would each have looked fine in review - fixtures covering a regex while the file-walk was never called; a self-test asserting only that nothing crashed; a loop over zero items reporting success; one that printed "Skipped is not passed" and then returned zero on the next line.

The countermeasure. Mutate the checker, not the fixture, and require the self-test to go red. Make it mechanical: make self-tests here runs a declared mutation per self-test and fails if one does not fire, and a self-test arriving without a declared mutation fails by construction. It caught itself on its first run.

7a. The method needs an executor, and the executor needs an invocation

The strongest instance in this repository, and the one that generalises furthest. A project can adopt a rigorous method, write it at the top of the file every agent reads, build the tool that enforces it - and never run the tool over everything.

Instance. CLAUDE.md's first rule here is "done is a command that exits zero", and 757 such commands sit in the backlog across 466 tickets. The tool that executes them is scripts/check_tickets.py. make tickets invokes it three times: --self-test, --docs-check, and --since main. Only the third executes anything, and only for tickets changed since main - which on main is the empty set. The full run takes over ten minutes and is in no target and no CI step.

So nobody found out that 63 of 598 criteria are red today, 44 of them pytest exit 4, which means the test file does not exist. Or that, of 80 sampled and attacked by reverting what each claims, 46 could not fail at all (DOC-075).

The purest single case: a criterion reading test "$(grep -c role .../user.py)" -eq 0. git log -S role over that file returns nothing - the string has never been there in any commit. It passed on the day it was written and every day since, and it was never checking anything.

The countermeasure. Three separate questions, and a method needs all three answered: is there an executor, does it have an invocation that covers everything rather than a diff, and does it run on a schedule somebody reads. An empty input set reports success, so a checker scoped to a diff is green on a quiet day by construction. And when the full run is too slow for the fast path, that is a scheduling decision to make explicitly - not a reason for it to have no invocation at all.

7b. "Covered" and "covered by what actually runs" are different sets

Instance. 70 deliberate mutations in this repository's auth, privacy, billing, egress and file-handling code: 57 killed, 13 survived, a kill rate of 81% (OPS-126). That is the first measured answer to whether the green suite is informative, and 81% is a real number rather than a reassuring one.

The split of the 13 is the finding. Six die the moment a database and the postgres suites are in play - the test exists, it is good, and the gate does not reach it. Six survive with a database too and are asserted by nothing. One could not be measured because its baseline was already red.

So roughly half the apparent coverage gap was not missing tests. It was tests nobody runs. Separately: make test-postgres is red on main with nine failures, every one an erasure, export or account-deletion test, while a CI job exists that runs them.

The countermeasure. Measure the suite under the selection your gate actually uses, not under the most generous one. A test excluded by a marker is a test that does not exist for the purpose of catching a regression tonight. And when a survivor turns up, re-measure it under a wider selection before filing it - the agent that produced this number had two false survivors from selections that missed a test naming the mutated function, and recorded the rule it learned: a survivor from a narrow selection is a fact about the selection.

8. Amplification of zero is zero

Gates, reviews and lenses multiply whatever detection already exists. They do not create it. A hundred passes over a claim nothing checks is a hundred confirmations of nothing.

Instance. Seventy-three review passes over this repository found roughly nothing that mattered. The owner, opening the product in a browser for five minutes, found eight defects. A separate measurement put review amplification at 1 finding to 75 words in one case and 4 to 8 in another - the ratio is a property of whether the underlying check was real.

The countermeasure. Before adding a review pass, ask what it would detect that nothing currently detects. Prefer one new executable check to three more readings. And walk the product: a browser on the real deployment found in two minutes what the instruments had missed for three days.

8a. Agents reason from documentation about the environment, and cannot check it

An agent can verify a claim about the code: the file is there or it is not, the function exists or it does not, and this repository has gates that check both. Claims about the environment are different. "CI runs this", "the hook catches that", "the runner has a database" - an agent has no cheap way to test any of them, so it repeats them. And unlike a claim about code, nothing rots more quietly, because it was true when it was written.

Instance. .githooks/pre-commit printed CI runs both checks over the whole tree regardless when it could not find ruff. On 2026-09-16 two agents each hit that path, each reported it faithfully, and each concluded their lint was covered. GitHub Actions had started no job since 2026-09-14 - 155 consecutive runs, every job refused in under four seconds for a failed payment, zero steps executed. Two lint errors reached main inside merged work, and the coordinator repeated the same false claim in a commit message before checking the runner (OPS-131).

Worse, in sequence: the last run whose jobs actually executed was already red, 5 failed, 116 passed, on the erasure tests. So five failures were reported and unread, then four more landed after the runners stopped and were reported by nothing (CORE-075).

The countermeasure. Two halves.

A tool must not promise a downstream it cannot see. A hook that cannot run a check should say "nothing else has checked this, run it yourself" - true whatever CI is doing - rather than naming a system whose state it has no access to. Every such sentence is a claim with no enforcer, which is the thing docs/PROCESS.md requires of rules and nobody requires of reassurances.

And check the runner, not the config. .github/workflows/ci.yml was correct the whole time: right triggers, no continue-on-error, no path filter. Reading it would have told you the job should run. Only gh run view showed zero steps executed, which is a job that never started rather than a job that failed. A green-looking pipeline and an absent pipeline are indistinguishable from the configuration alone.

8b. Shared infrastructure makes every agent's environment differ from yours

In a multi-worktree arrangement, some things are per-worktree and some are shared, and the split is rarely written down. Tooling written by somebody working in the main checkout quietly assumes the main checkout's layout.

Instance. core.hooksPath is an absolute path into the main checkout, so every worktree runs the same hook - while that hook looked for ruff at ./.venv-verify/bin/ruff, relative to whichever tree invoked it. make setup builds that virtualenv once, in the main checkout. So the lookup resolved for a person and missed for every agent, falling through to a shim that resolves and does not run. Fixed by also resolving $(git rev-parse --git-common-dir)/.., which is the plumbing that answers the main checkout from inside a worktree.

The same asymmetry bites twice more here. An agent editing .githooks/ in its own worktree cannot test its change, because the hook that runs is main's - so verifying it needs git -c core.hooksPath=$PWD/.githooks. And the git stash stack is shared across every worktree, which is why this repository forbids git stash outright after two agents popped each other's work.

The countermeasure. Write down which parts of the environment are per-worktree and which are shared, and resolve shared tooling through git plumbing rather than through a relative path. When a tool reports that it could not find something, have it say where it looked - this hook did, which is the only reason the cause took two minutes to find rather than an afternoon.

8c. A conclusion reached honestly decays into a fact without its derivation

The failure modes above are mostly about being wrong. This one is about being right and losing the record of why, which is more dangerous because nothing about it feels like a mistake.

Instance, 2026-09-17. The duchovni-net session told this one that a Cloudflare API token held here could edit the owner's MX and DMARC records, and wrote it as a fact in two of its own documents. It was almost certainly correct. But it had not measured it - it had inferred it, soundly, from the scopes it had recommended minting plus the knowledge that Cloudflare cannot scope a token below a whole zone. Asked, it found the inference in its own files presented as an observation, and separated them: measured is that the token exists, answers and returns exactly one zone; inferred is everything about what it can do.

Its own description of the mechanism is the best one either session produced:

I had reached the conclusion honestly and then stopped distinguishing how I got there, which is the failure that looks exactly like knowing something.

This session did the same thing in the other direction the same day: it wrote that the token's scopes were unverifiable, which was true, and then had to resist absorbing the other session's confident inference simply because it agreed with it.

Why an AI-led project gets this worse than a human team. A person carries the feeling of having looked something up, and it fades noticeably. An agent writes a summary, the summary is all that survives into the next context, and a summary has no slot for "inferred" unless somebody built one. Every compaction, every handoff and every report is a step where the derivation can fall away while the claim travels on - and the claim travels because it is useful.

The countermeasure has two halves and the first is not enough on its own.

Separate measured from inferred in the artifact, not in your head. A document that says "verified: X. Inferred from Y: Z" survives a handoff; a document that says "X and Z" does not, and no amount of having known the difference at writing time helps the next reader.

And attribute a claim you believe. That is the hard half. Attributing a claim you doubt is easy and costs nothing. Attributing one you are confident in feels like hedging, reads like distrust of a colleague, and is the only version that protects anything - because the claims that do damage are the ones everybody agreed with. BLOCKED-ON-YOU.md records this token's mail-record reach as one session's assertion rather than as a finding, while both sessions believe it is true.

The standing example of the cost is in this repository's own history: a backup was described as working for weeks, honestly, by people who had every reason to think so, and OPS-140 found that nobody had ever restored one. Nothing was dishonest at any point. The derivation had simply stopped travelling with the claim.

9. The factory outruns the product, and nothing notices

An agent improves the thing the tickets are about. In a template-and-product arrangement the tickets are about the template, so the template improves and the product - the only thing a customer touches - receives none of it.

Instance. Measured 2026-09-16 by diffing every kit Python file against the one live product: 68 identical, 54 divergent, 108 absent from the product. Absent included the error pages, the error tracking, the egress seam and the security-headers middleware. The propagation command, saas update, exists and is documented in five places, and had never been run. Two consecutive sessions measured the ratio of factory files touched to product files touched at 24:1 and then 89:1.

The countermeasure. Measure the distance between factory and product on a schedule, and treat the propagation path as a walk that must be exercised, not a command that exists. A capability the product does not have is not a capability.

10. An agent's report is not the work, and often not accurate about itself

Agents summarise their own runs, and the summary drifts from the branch - not from dishonesty, but because the report is written from memory of intent while the branch is a record of what happened.

Instance. One agent's report was wrong about its own commit count and conflict count, and corrected the coordinator's claim of nine commits to six. Another's count of self-tests corrected the coordinator's 25 to 26, because the coordinator had grepped one spelling. The coordinator, separately, reported "all gates green" after running two of fourteen layers, and named fourteen gates honestly while the Makefile documents fifty-four.

The countermeasure. Verify the claim you are about to repeat, against the artefact rather than the report: git log --oneline for the commits, git rev-list --count for the number, the command's own output for the result. Spell the thing a second way before writing that it is absent - a search that found nothing is a fact about the search.

The shortest version

Six habits, each of which paid for itself in this repository:

1. Falsify every check you add. Break the thing, watch it go red.
2. Count effects, not attempts. Zero substitutions is a failure.
3. Assert the wrong value's absence, not just the right one's presence.
4. Name the class, and make the last assertion cover it.
5. Walk the product in a browser, on the real deployment, regularly.
6. Check the artefact before repeating a report - including your own.
7. Ask what invokes your checker, and whether its input set can be empty.
8. Measure under the selection your gate runs, not the generous one.
9. Never promise a downstream you cannot see - and check the runner, not the config.
10. Resolve shared tooling through git plumbing, because an agent's tree is not yours.
11. Separate measured from inferred in the artifact - and attribute a claim even when you believe it.