Diffing prompts the way you diff code

Ask an engineer where their prompts live and the honest answer, most of the time, is a chat window and a text file called notes.txt. Ask which change made last week’s output worse and there’s no answer at all, because there’s no diff to look at.

That’s the whole gap. Everything else about prompt engineering is downstream of the fact that the artefact isn’t under version control, so none of the tools you’d normally reach for apply.

Prompts are source, so treat them as source

The fix is unglamorous. Put the prompt in a file in the repository, next to the code that uses it. Give it an extension your editor won’t fight you over. Then every mechanism you already have starts working for free.

git log tells you when it changed. git blame tells you who changed it and, if the commit message was any good, why. A pull request makes a prompt change reviewable by someone other than the person who wrote it, which turns out to matter enormously, because prompts fail in ways that are obvious to a reader and invisible to the author.

The thing that surprised me most after moving to files: how much dead text accumulates. A prompt edited live in a chat window grows by accretion. Nobody deletes a line from a thing that seems to be working. Put it in a diff and the redundant paragraph you added in March becomes visible immediately, because someone has to read it in review and ask what it’s for.

The two things that make a prompt diff meaningful

A diff on its own only tells you the text changed. To learn anything you need two more pieces, and both are cheap.

Pin the model version in the same file, or right next to it. A prompt is only half the system; the other half ships updates you didn’t ask for. If your prompt file doesn’t record which model it was tuned against, then six months from now a regression could be your edit or could be a model change, and you will have no way to tell. Write the version down. It costs one line.

Keep a fixture set. Not a test suite, nothing that elaborate: five to ten real inputs, saved, with the output you consider correct. When you change the prompt, run the fixtures and read the differences. That’s it. This is far below the sophistication of a proper eval harness and it catches most of what a proper eval harness catches, because most prompt regressions are not subtle.

The failure mode without fixtures is specific and universal. You tweak the prompt to fix the case that annoyed you this morning, it fixes that case, you ship it, and three weeks later you notice something else got worse. With ten saved inputs you’d have seen it in ninety seconds.

Change one thing

This is the part everyone knows and nobody does, and prompts make it worse than usual because editing feels free.

A prompt edit that changes the tone instruction, adds an example and reorders two constraints is three experiments in one commit, and when the result improves you’ve learned nothing about which of the three did it. Worse, if two changes moved in opposite directions you’ll conclude the whole edit was neutral and keep both.

Small commits, one change each, fixture run between them. It’s slower and it’s the only way the knowledge compounds. Six months of one-change-at-a-time leaves you with actual understanding of what your prompt needs. Six months of bulk edits leaves you with a long prompt nobody dares touch.

What review catches

Once prompts are in pull requests, a category of problem starts getting caught by other people, and it’s a consistent category.

Instructions that contradict each other, usually because they were added months apart to fix different things. Examples that demonstrate the opposite of what the surrounding text asks for. Constraints written in the negative, which models handle poorly and humans read straight past. Elaborate formatting requirements that the downstream code immediately strips.

None of those are visible to the person who wrote the prompt, because they know what they meant. All of them are visible in about thirty seconds to a colleague reading it cold, which is the same reason code review works.

The honest limit

None of this makes prompt behaviour deterministic. You can have perfect hygiene and still get different output from identical input, and no amount of version control fixes that.

What it fixes is the attribution problem. When something degrades, you want a short list of suspects rather than a shrug, and a repository gives you one. That’s a smaller claim than the phrase “prompt engineering” usually implies, and it’s the part that actually holds up on a team.

More field notes on prompt engineering

This piece is one entry in a running series on how AI coding tools change day-to-day engineering work. For more practical notes on prompt engineering specifically, browse the full set at /blog/tag/prompt-engineering/. For the wider view across every tool in the stack, the AI coding tag collects the whole archive in one place.