Why We Run Coding Agents in a Loop

An AI coding agent that runs for an hour fails differently than one that runs for a minute. The short session either works or visibly doesn't. The long one produces something plausible, well-structured, and confidently described — and the interesting question is whether any of it is true.

We build a code index, so our first instinct was that better retrieval would fix this. It doesn't. Retrieval fixes wrong answers about the code. It does nothing about an agent that quietly stops working on the task you gave it, or one that reports success it never checked. Those are process failures, and no amount of context solves them.

This post is about the two failure modes we kept hitting, the discipline we built to stop them, and five specific times that discipline caught something in our own release history. The examples are from real commits in a release we shipped this week. Where I'm generalizing rather than reporting, I say so.

The two ways long sessions fail

Drift. The work becomes something adjacent to what you asked for. Rarely a dramatic swerve — usually a slow substitution of an easier problem for the real one. The most damaging version isn't scope expansion, which is at least visible in a diff. It's scope contraction: the task narrows to the part that was tractable, the summary describes that part accurately, and nothing in the output announces that the rest was dropped.

Unverified done. "Done" gets declared on the basis of the change looking right. The tests were written and they pass. The code reads correctly. Neither of those establishes that the thing works for the person who will use it, and the gap between "my tests pass" and "the customer path works" is where our worst bugs have lived.

What makes both hard to catch is that a drifted session and a faithful one produce equally confident summaries. You cannot tell from the output which one you got.

The loop

The response is a fixed order with an exit condition on every phase:

diagnose  →  research  →  design  →  plan  →  implement  →  verify  →  ship gate

The gates are the entire product. Each phase exits on a condition, not on a feeling:

• Diagnose — name the actual defect, which is a different object from the reported symptom.
• Research — establish what's true and how you know. Findings go somewhere durable, not into chat history.
• Design — only when there are genuine unknowns: a schema, a contract, an assumption about code nobody has read.
• Plan — ordered steps, each with a checkable outcome.
• Implement — one step at a time. No step begins before the previous one is verified.
• Verify — evidence: a named test count, a command's output, a probe of the surface you changed.
• Ship gate — the loop stops. Committed and verified locally is terminal. A human decides whether to release.

If we could keep only one of those, it would be the ship gate. Everything above it is recoverable on a local branch.

Five things this caught

All five are from the commit history of one release.

1. Built, tested, and unreachable

We built an installer for a new capability. It was correct. It had unit tests proving it wrote the right files given a directory. It shipped in the package.

Nothing called it.

The commit that fixed it is titled "the call sites Track A could not add," and the code comment we left is the most useful sentence in the release:

it built install_workflow_layer and proved it writes the right files given a directory, but nothing invoked it, so no customer running cix install ever received the layer.

Every test was green. Coverage was fine. The feature did not exist for anyone.

2. A rollout switch wearing an entitlement's clothes

The same capability was gated by an environment variable. Convenient during development, and wrong: a local variable that can turn a paid feature on is not entitlement, it's a rollout flag that happens to be reachable by anyone.

The fix made the variable subtract-only — a kill switch that can stop an entitled user from receiving the feature and can never grant it — and moved the actual decision to the server, which fails closed on every uncertainty: no credentials, unreachable API, non-200, malformed body. The commit says it plainly: "the env var is a kill switch; the SERVER decides entitlement."

3. Verification that certified the wrong thing

Our release acceptance harness could certify a build and print a clean pass — for a commit other than the one under test. Commit title: "the gate could certify the wrong commit and still print a clean pass."

Everything downstream had been inheriting a guarantee that was never made. A related fix in the same release — "a 2xx alone can never mean 'denied' for a mutating probe" — was a probe reading any successful response as proof of refusal, when a 2xx on a mutating request is closer to proof of the opposite.

4. A guard that refused instead of clobbering

This one we hit live, this week, on our own machine.

Running the installer produced FileExistsError and installed nothing. The cause turned out to be correct behavior: the installer hashes every file it is about to write and proceeds only if the path is absent, byte-identical, or recorded in a prior install receipt. Seven target paths were occupied by hand-maintained files it didn't own, so it refused all seven rather than overwrite any.

The guard worked exactly as designed. The message was the defect — it printed only the exception type, discarding the list of seven files the exception was carrying.

So the safety property held and the reporting property failed, and from the outside those are hard to tell apart. We spent twenty minutes establishing that the installer was fine. Someone less patient would have concluded it was broken and worked around it, which is the rational response to a refusal nobody explained.

5. Tests that can't fail

While writing the change this post accompanies, I published a documentation section that had been deliberately withheld. A test asserted that the set of withheld pages was non-empty — a reasonable-looking guard that had the effect of requiring something to stay unpublished forever, or the suite would go quiet.

Publishing everything would have made the surrounding assertions iterate an empty set and pass while checking nothing. The fix was to test the gate against an injected fixture rather than against whatever the real content happens to be. The repository already had this idea, in another test file: "A scan matching nothing would pass the test above forever."

Making the gate injectable then broke the function that lists published pages, which passed the predicate straight to .filter — so the array index arrived as the new second argument. The suite caught it on the first run, about a second after the edit. The change had looked right.

Where the state lives

A loop that runs across sessions needs memory that isn't the conversation. Chat history is a bad database: long, unsearchable, and ordered by time rather than topic. Transcripts may well survive on disk, but that isn't the same as a dependable record — the next session doesn't get them by default, and nothing in them separates a decision that stuck from an idea that was raised and dropped four messages later.

So the phases write to durable places — investigations to research, shaped work to designs, sequencing to plans — stored with your account rather than in files someone has to carry. The practical test is whether a session with no memory of the last one can pick up the work. That's the common case, not the edge case.

In practice that means writing for a reader who knows nothing, which costs a sentence at a time.

When not to use it

Everything above this point is something we watched happen and can point at a commit for. What follows is what we've settled into since, and I'd hold it more loosely.

The loop is overhead. It earns its cost when work can go wrong in ways you wouldn't notice: multi-session tasks, code you haven't read, an unestablished cause, a contract or schema change, or a session running with fewer interruptions than usual.

It is not worth it for a typo, a copy tweak, or a one-line fix to code you just wrote. Running full ceremony there generates artifacts nobody reads and slows down work that was never at risk. Treating every task as loop-worthy is its own kind of drift.

Adopting it

In the order we'd suggest, for the same reason — this is preference, not measurement:

1. Start with the ship gate. If you change nothing else, stop letting the agent push. Terminal state is committed and verified locally; a human decides to release. This is the highest-value change and the cheapest.
2. Then evidence-before-done. Require a named artifact for every completion claim: a test count, a command's output, a probe result. "It looks right" stops counting.
3. Then the phase order, on one real task. The discipline is more convincing after you have watched it stop something.
4. Add design only when there are unknowns. Skipping it for small work is correct; skipping it out of impatience is how you get three days of work on a wrong assumption.

None of this makes an agent smarter. It makes the failures visible, which turns out to be the part that was missing.

The loop, and the research, design, and planning workspace behind it, ship with cix. See the workflow documentation for how the pieces fit together.