Writing acceptance tests first, agents second

The failure isn’t the one everyone worries about. Agents rarely produce code that doesn’t work. They produce code that works, passes review, and does something adjacent to what you needed.

You asked for pagination and got pagination that breaks on the last page. You asked for retry logic and got retry logic without a cap. Both compile. Both look right in the diff. Neither is what you meant, and the reason is that your prompt was the only specification in the room, and prose specifications have been failing this way since long before any of this.

The test is the specification the agent can’t misread

Write the acceptance test first and the shape of the whole interaction changes.

The agent now has an unambiguous target. Not “handle pagination correctly”, which is an invitation to interpret, but a function call with an input and an expected output. There’s no room for a plausible alternative reading, because the reading is executable.

More usefully, you get a stopping condition. This is the part people underrate. Without a test, “done” is a judgment call made by a tired human reading a diff, and tired humans approve things. With a test, done is green, and the agent can iterate against it without you in the loop for every attempt.

And you find out whether you understood the requirement. Writing the test forces the specifics that prose lets you skip. What happens on an empty result set? What’s the behaviour on the last page? Is the cap inclusive? Half the time I discover I don’t know, which is exactly the discovery I want before an agent writes four hundred lines premised on my not knowing.

Acceptance, specifically, not unit

The word matters here, because the obvious mistake is writing the unit tests first.

Unit tests encode a design: this class, this method, this collaborator. Write them before an agent starts and you’ve pre-committed to a structure, which is the one part you probably want to leave open. Worse, an agent handed a set of unit tests will produce exactly the shape they imply, including the shape you’d have rejected if you’d thought about it for another minute.

An acceptance test encodes behaviour at the boundary. Given this request, this response. Given this file on disk, this output. It says nothing about how, which leaves the agent free to solve it and leaves you free to disagree with the solution without invalidating the test.

The rule of thumb I use: if the test would need rewriting after a refactor that changes no behaviour, it’s the wrong test to hand an agent.

What this does to the prompt

Something slightly surprising happens once the test exists. The prompt gets shorter and better.

Half of what goes into a long prompt is an attempt to pin down the requirement in words, which is precisely the job the test now does more reliably. What’s left is context the test can’t carry: which module this belongs in, which existing helper to reuse, which approach you’ve already tried and rejected.

That residue is genuinely useful prompt material, and it’s usually three sentences. The prompts I write now are much shorter than the ones I wrote a year ago and get better results, not because I got better at prompting but because most of what I was writing was a badly specified test.

The honest failure modes

Two, and both are worth naming because the technique gets oversold.

An agent will write code that passes your test and is otherwise poor. Tests constrain behaviour at the boundary and say nothing about whether the inside is a mess. You still have to read the diff. What the test buys is that you’re reading for quality rather than for correctness, which is a much easier read and one you’re far better at.

And an agent will occasionally satisfy a test by cheating: special-casing the exact input, or worse, quietly weakening the test. That last one is common enough to build a habit around. Before you accept anything, check whether the test file changed. If the agent edited both the implementation and the test in the same turn, read the test change first and read it properly.

The version of this that costs people real time is the skipped test that nobody notices for three weeks. Run the suite yourself, look at the count, and treat a shrinking number of assertions as the alarm it is.

Where it doesn’t apply

Exploratory work, obviously. If you don’t know what the thing should do yet, writing a test first is a way of pretending you do, and the agent is genuinely useful for the exploration phase precisely because it’s cheap to throw away.

The discipline starts when you know the requirement. At that point the test is faster to write than the paragraph describing it, harder to misread, and it’s still there in six months when someone changes something nearby.

More field notes on AI coding workflows

This piece is one entry in a running series on how AI coding tools change day-to-day engineering work. For more practical notes on workflows specifically, browse the full set at /blog/tag/ai-workflow/. For the wider view across every tool in the stack, the AI coding tag collects the whole archive in one place.