The product is not the experiment. The factory is.

I run engineering for a company where a few hundred people build, secure and operate a cloud platform that healthcare organisations depend on. The question in front of every engineering leader right now is not whether AI changes how software gets made — it’s what actually holds when it does. Which parts of the lifecycle can agents carry. Where quality really comes from once a machine writes the code. What has to be true before you can move fast without quietly becoming loose.

You cannot answer those from a keynote, a vendor demo, or a pilot that someone else runs. You answer them by building something real, end to end, and living with the consequences.

So I built a software factory on the side.

What I actually want to learn

What agents can carry, and where they stop. Not the demo version — the version where an agent works inside a real repo, with real migrations, real tenants and real production data, and has to be right on a Tuesday night with nobody watching.

Where quality comes from when a machine writes everything. Reviews, tests, gates, evals, contracts between components. Which of those matter more when the author is a model, which matter less, and which turn out to be theatre. My working answer so far is that the interesting work moves upstream — into what you specify and what you verify — and that a green test past the real code path is worse than no test at all.

What lets you go fast. Not raw generation speed, which is the least interesting variable. The things that actually set the pace: how quickly a change can be proven safe, how much context has to be rebuilt each time, how many decisions bottleneck on one person.

What green teaches you that brown cannot. A new project is the easy case, and I knew that going in. It’s also the honest one: there’s nowhere to hide behind legacy. The useful part is what happens next — because after a few months, a green project becomes a brown one. It grows migrations you have to live with, decisions you’d make differently, tests that outlived their reason. Watching that transition happen in fast-forward, in a system I understand completely, is worth more than any greenfield lesson.

And the part I expect to be hardest: change management. This is the piece I’d emphasise over everything else, and the piece most of the industry’s writing skips. Making a factory work with five agents and one person is a technical problem. Evolving the software factory of a few hundred people — the whole product development lifecycle, from an idea to thousands of customers using it in production, at the quality and security bar that healthcare demands — is overwhelmingly a people problem. Roles change. Craft that people are proud of changes shape. Trust has to be earned by evidence rather than announced in a deck. Anyone who thinks the hard part of this transition is the tooling has not tried to lead it. I want to arrive at that conversation having done the work myself, with scars and specifics, not slideware.

The terms I set myself

My own time. Evenings, mostly. I start agents at night and they work while I sleep; I read what happened over coffee. Some weekend mornings, some late nights. It fits around a demanding day job because it has to.

My own money. Every model call — Claude, Codex, Cohere — and every piece of infrastructure: GitHub, Vercel, Neon, Cloudflare and a handful of other services. Nothing is expensed, nothing is subsidised, no vendor is giving me credits. That matters more than it sounds. When the bill is yours, you find out fast which parts of this are economically real and which are a demo that happens to be free. The build cost is published in these posts, monthly, because a factory whose economics you can’t state is not a factory you understand.

In public, including the wrong turns. Every sprint gets written up here — what shipped, what broke, what I got wrong. I’ve started enough projects where an honest sprint diary would have been more useful than the eventual launch post.

Why it needed a real product

An experiment about quality needs something whose quality can actually be judged. A toy passes every test you write for it, because you wrote them for the toy.

So the factory needed a real use case: a product with real users, real data, real consequences for getting it wrong, and someone who would notice immediately if it degraded. The most reliable such person available to me was me — and the most reliable source of unglamorous, unforgiving requirements was my own household.

That’s the other half of this story, and it has its own post.