The 3x3 Rule of AI Assisted Software Development

The goal of AI Assisted Software Development, as I see it, is to figure out how to make a repeatable, consistent process out of building software with LLMs. To Escape the Claud-ottery and know what you're coming back to each time you send you little AI minion off to do its work.

To work towards this, I always have 3 triplets floating around in my head:

Plan, Plan, Plan
Constrain, Constrain, Constrain
Review, Review, Review

Why triplets? Because when working with AI you're should always be actually working at three levels: The Project / The Process / The meta-process. The same thing happens in normal software development, but in the case of AI Assisted software development the cycles through the loop are so fast, that there's no time to adjust in-flight, You need to have each nailed down or you'll be disappointed with the results.

Plan, Plan, Plan

AI still screws up a lot. It's actually a bit of a horror show watching it work as each individual step is often riddled with errors. With a good process (what we're talking about here), you'll end up with something that works when it's all said and done, even if each individual step would give you indigestion.

The key thing to note is that generally, AI is perfectly willing to tell you exactly how it's going to screw up ahead of time - just tell it to write up a plan first. At the project level this should involve asking it to create a markdown document with its plan. I generally create 3 documents in this order: A high level architecture document for the project, a phasing document, and then a single phase design document (rinse and repeat for each phase). I've ended up with individual skills for each of these that I've extracted after a few months of writing the same thing over and over again.

At the process level you need to tell it how to write and test the software it's building. This ties into the Constrain, Constrain, Constrain a bit in the specifics, but at a high level you need to set it up with a process otherwise it'll do whatever it feels like that day (all hail the stochastic gods). I point it towards a TDD-style process that is heavy on linting and puts user gates in appropriately (Architecture decision? Ask me). For small projects, this just means updating CLAUDE.md - for larger projects, I like to create a separate AGENT_IMPLEMENTATION_SPEC.md file and a skill that points towards that file for project-specific details.

At the meta-process level this means thinking about the explicit steps in the development process you want to go through. For me, at the individual phase level, it involves two user-gated parts:

I'll go through the specifics in another post, but the idea is creating a concrete plan for how the development should happen means you can run each step manually for a bit before creating skills that start to bundle these steps together.

Constrain, Constrain, Constrain

The first constrain rule to let the LLM do no harm. To achieve this you need to keep it away from the important stuff. The easiest approach to this is to use a devcontainer (a development-specific docker container) and keep your Agent helpfully contained therein. VSCode and Zed work seamlessly(-ish) with development containers. I deploy to AWS and I want to be able to test that via the Agent, so I'll add credentials into the container for an AWS staging environment - things can obviously still go horribly wrong from a cost perspective ("Claude: I trained a new model for you on 10 p5.48xlarge overnight..."), but at least production is safe. Keep GitHub credentials out of the container (or create a read-only user) and push from your home terminal.

The second constrain rule is to try your darnedest to not let the LLM write code that doesn't work. This means using a typed language - like Typescript instead of JavaScript, or a compiled language like Go, or adding type hints to a dynamic language like Elixir. It means defining and running the appropriate linting commands for each step. It means telling the LLM to write tests and to not declare a phase finished until all the tests pass and test coverage is > 85% (or some appropriate, arbitrary percentage). The key is to give it tools tools that help its feedback loop to make the right decision.

The last constrain rule is the meta-feedback constraints. I like to think of LLMs as incredibly fast typing Jr. developers who have an encyclopedic knowledge of StackOverflow, but get their mind wiped every night. The only way to get them to remember stuff is to tattoo them Memento-style with the project-specific info they need.

This means updating AGENTS.md or CLAUDE.md or supporting document files when the LLM struggles with a specific issue. Otherwise it will struggle with it again in the future. This ties into the next triplet - Review, Review, Review.

Review, Review, Review

At the project level - review means two things: an automated review followed by a manual review.

For the automated review, it's important to make sure the review isn't influenced by the context of the implementor - so having the review use a fresh context is important.

Secondly, it's important to review at three levels:

The Code - what the agent actually wrote, looked at from a code perspective. This is the most obvious as it's the Agentic equivalent of the pull request. Building a review sub-agent that pulls from a project-specific code review criteria is a straightforward approach. I've end up with two separate code review agents - a general one using my own skill, and one that applies the claude security-review skill.

The Functionality - what was the agent building and does it actually work? (hint: probably not, or at least not in the way you intended). The functionality needs to be exercised outside of the tests and from a user perspective. This means exercising APIs via curl and checking the database. Exercising browser functionality via playwright-cli and exercising mobile app functionality via simulators. This should be done with a subagent that only has the spec of what was built. You'll still likely need to review manually to tackle UX craziness, but this is a good first gate.

The Process - where did the agent stumble? Use a subagent to review the transcript of the implementor agent to see where it burned tokens it doesn't have to burn in the future. Unlike a human that learns not to make the same mistake the next go around, unless you put an explicit review of the process that looks at the transcripts of the build process your agents won't "learn" in the normal sense.

When the review sub-agents agents have finished their work, each of which creates a written artifact, I have the main thread run a fix agent that takes the code review and the functionality review and fixes it. If something comes up that it can't make a decision on - that's when you'll need to step back in and make a decision.

I keep the process review separate and run it out of band from the build loop, as many times it's better to let a few of these retros accumulate before applying them to get closer to the "rule of 3" and not end up with a bunch of 1-off cruft instructions that won't apply again. Agents are sticklers for process however, and Claude starts to get angry when there are too many unapplied retros.

With the overall approach from this post, usually it's just a few tweaks to get to the desired outcome, but sometimes there's still a "Oh, I forgot to specify something I took for granted" and you'll need to throw out the code and run the loop again.

The good news is that by tackling the process and meta-process at each run through the loop, your agents should be improving with each run.

Extra Credit: the above process, unfortunately is primarily additive. You're adding new rules, new steps and skills as you go along. This will work well for a time, but at some point you have to step back and look at what you've created. Many times you'll end up a repetition and bloat in your project specific spec files and those will need to get a haircut, or some of your internal documentation is out of date and will need to be rebuilt. You don't need to do this on every loop, and on small projects you may avoid it entirely, but if you are eating a 100k of context before any work is being done your do for a visit to the context barber.