Running the Factory

The first thing I built this system to do was fail on purpose.

Pull request #3 in the repo is titled DO NOT MERGE, gate verification (planted hardcoded API key). It puts a fake credential in a source file, in plain sight, on a line written to slip past every automated check. The code is valid. The tests pass and CI goes green (CI is the battery of automated checks every change runs through before it can ship), which was the point: to prove that CI alone cannot catch a leaked secret.

The reviewer caught it. The reviewer is a second AI, OpenAI’s Codex, that reads every change against the repo’s written rulebook, and it flagged a P0, the most severe class of finding, against the repo’s own rule on committed secrets. Then it added the part I keep coming back to: even as a fabricated test key, merging the pattern trains the app to ship credentials in code.

I closed it unmerged and kept the technique: a planted bug, a canary, whose only job is to prove the gate can still catch one.

What the thing is

Anyone can generate an app now, which means an app proves less than it used to. So I want to be precise about what Intervals is. It’s a real, deployed SaaS: Next.js, Supabase, Postgres with row-level security, Drizzle migrations, shipping to production on Vercel through GitHub Actions. It forecasts delivery with a 10,000-run Monte Carlo simulation: scope and pace in, “when will this be done?” answered as a distribution with confidence bands, and a record of whether the forecast held. I’m the primary user. I’ve also never shipped production code in my life.

I want to be precise about where the gap actually is, because “not technical” is too blunt and lets me off too easy. I have a BS in mechanical engineering. I’ve written plenty of embedded C for microcontrollers and test assemblies, which is real programming, and it is nothing like standing up a multi-tenant web application with authentication, a database that strangers’ data sits in, and a deploy pipeline. Firmware on a test rig fails in front of me, on my bench, where the worst case is a wasted afternoon. Production software fails in front of a stranger who trusted it. I’ve done the first and never the second, and every gate in this system exists because of that sentence.

So I don’t write the implementation. I decide what gets built, in what order, against what constraints, and then I run a loop of AI agents that writes it, reviews it, and tells me when it’s wrong. That loop is the factory of the title. My job is the judgment and the gates: the spec, the architecture decision, the call on what merges.

That only works if the output is good. Someone in my position shipping a vibe-coded toy proves nothing at all. Shipping software with verified per-user data isolation, an independent review gate, and a secret scanner in CI is a different claim: that the discipline can come from the operator even when the typing doesn’t. What follows is the machinery that makes the claim checkable, and where the checking still falls short.

The loop

I label well-scoped tickets in Linear (the issue tracker) claude ready. Overnight, a scheduled routine picks them up, and each one gets its own branch, a draft pull request, and an independent review from Codex against the rulebook, AGENTS.md, which holds my standards for the project ranked P0 to P2, severe to minor. The agent drives the review-and-fix loop until the review is clean, moves the ticket to In Review, and stops. It never merges.

In the morning I run /integrate. It triages the night’s pull requests, rebases the ready ones, re-runs CI, gets a fresh review pass, and then I approve. Merge authority is always mine.

The rule underneath all of it is one sentence: green CI is necessary, not sufficient.

Pull request #3 is that sentence as an artifact. The code was valid, so every automated check I owned came back green, and the thing was still wrong in the way that mattered most. Tests passing means the code does what it says. Whether it should say that at all is a separate question, and that question is the judgment I’m supposed to be providing.

So nothing reaches main until CI is green and an independent review is clean. The second condition is where my judgment lives, and the factory’s job is to keep enforcing it overnight while I’m not watching.

What the gate has caught

138 merged pull requests so far, every one squash-merged and tracked to a Linear ticket. The reviewer has raised 659 inline findings against them: 11 P0, 248 P1, the rest lower.

Ten of those eleven P0s landed on a single pull request, and none of them were code. They were a specification, for organizations and role-based access, reviewed before anything had been implemented. One of them, in the reviewer’s own words:

A USING/WITH CHECK policy that only requires member access permits a user who belongs to orgs A and B to update a tenant row’s org_id from A to B: the old row passes USING, and the new row passes WITH CHECK. That exposes A’s artifact to every member of B and changes its owner tenant.

In plain terms: a user who belonged to two organizations could move a record from one to the other and expose the first org’s data to everyone in the second. A cross-tenant leak, found by reading a document. The other nine on that PR are the same shape: an invitation flow that could quietly hand someone owner powers, a deletion rule missing its permission check, a provisioning function anyone could run. Every one of them would have been baked into the schema, the database’s structure, by the time anyone wrote code against it, and the schema is the expensive place to find out you were wrong.

The loop learns

When the review catches a real class of mistake, the fix goes two places. The instance gets corrected, and the pattern goes into a versioned docs/learnings/ library organized by failure class, which is the same material future tickets get written against. The next agent starts with the lesson already in hand, so every bug the gate catches makes that class of bug harder to reintroduce.

Fourteen entries so far. Two of them are about the reviewer’s own failure modes: the cases where Codex stops early, and the cases where it reviews a stale commit and reports clean. I think those two are the most valuable documents in the whole system, because a gate you never test is a gate you’re taking on faith. That’s why #3 exists, and why I still plant new canaries from a catalogue. If the reviewer waves one through, I find out at zero stakes, from a bug I put there myself.

What you can and can’t check

The Intervals repo is private, and I’m going to leave it that way. I’m not building this into something I sell, and given where my experience runs out I’d rather not invite strangers into a deploy path I can’t fully audit myself. The counts above come from its pull request history, which means they’re mine to assert and nobody else’s to verify. That’s a real limit on this piece, so I’m naming it.

What I have opened is the part that transfers. The factory repo is the loop with the product removed: the rulebook the reviewer enforces, the overnight routine, the canary catalogue, and the learnings library. Most of the machinery is wiring between services that has to live in your own accounts, so it’s a reference to read and rebuild in your setup, and cloning it won’t give you a running factory.

Read AGENTS.md first. It’s a hundred and thirty-five lines of soft judgment, the things I’d say in a code review if I were qualified to give one, written down hard enough that a machine can enforce them at two in the morning without me. If you only read one file in the repo, make it that one.

The open question is whether it keeps compounding. Fourteen learnings entries is a library; it isn’t yet proof that the curve bends. The number I’m watching is revision cycles to green: how many review-and-fix rounds a pull request takes before Codex passes it. If the library is doing its job, that should fall, because the agent inherits the lesson instead of rediscovering it. If it holds flat over the next fifty merges, I’m documenting mistakes rather than preventing them, and I’ll have to say so.