Prompt engineering is becoming a software engineering problem

The first wave of prompt engineering was about wording. You'd sit in a playground, add "think step by step," rephrase the instruction three ways, watch the output improve, and ship the version that looked best. It felt like a craft — part intuition, part incantation. And for a demo, it was enough.

Then the prompt went into production. It ran a thousand times a day against inputs no one had tested. Someone tweaked it to fix a customer complaint, and a week later a different customer complaint appeared. Nobody could say which change caused it, because the prompt lived in a text field in a console and had no history. That's the moment prompt engineering stops being wordsmithing and becomes a software engineering problem.

The claim of this post is simple: a production prompt is code. It has inputs, outputs, edge cases, regressions, and a blast radius. If you'd never ship a function without version control, review, and tests, you shouldn't ship a prompt that way either.

The anti-pattern: vibes-based prompt tweaking

Here's the workflow most teams start with, and many never leave:

• The prompt lives in a string literal, a config value, or worse, a text box in a vendor console.
• Someone notices a bad output, edits the prompt until that specific output looks better, and saves.
• There's no record of what changed, why, or what it might have broken.
• "Better" is judged by looking at one or two examples — the ones that prompted the change.

This is the equivalent of editing production code with no diff, no review, and a test suite of "I ran it once and it seemed fine." It works right up until the prompt is doing real work for real users, and then every change is a gamble. You fix the thing in front of you and pray you didn't break the ten things you can't see.

The fix isn't a cleverer prompt. It's the same engineering discipline you already apply to everything else.

Prompts as versioned artifacts

Start with the most basic thing: the prompt lives in the repo.

Not in a console. Not in a database row someone can edit at 2am. In a file, in version control, changed through pull requests, reviewed by a human. This one move buys you an enormous amount:

• History. You can see exactly how the prompt has changed over time, and git blame tells you who changed what and why.
• Review. A second person looks at the change before it ships. Prompts are subtle — an instruction that reads fine to the author often has an obvious failure mode to a fresh reader.
• Rollback. When a change goes wrong, you revert a commit instead of reconstructing yesterday's prompt from memory.
• Coupling to code. The prompt and the code that parses its output change together, in the same commit, reviewed as a unit.

Store prompts as structured files — a template with named variables, a version identifier, and metadata about the model and parameters they're tuned for. Treat the model name, temperature, and max tokens as part of the artifact, because a prompt that behaves well at temperature 0 can fall apart at 0.7. The prompt isn't just the words; it's the words plus the runtime configuration.

The eval harness

Version control gives you history. It doesn't tell you whether a change is good. For that you need evals, and an eval harness is the single highest-leverage thing you can build.

An eval harness is conceptually the same shape as a test suite:

1. A set of labeled cases. Realistic inputs paired with what a good output looks like. Pull these from real production traffic, not from your imagination — the cases you invent are the ones the prompt already handles.
2. A runner. Something that feeds each case through the prompt and collects the output.
3. A grader. Something that decides whether each output is acceptable.

The grader is where it gets interesting, because there are three tiers and you want to reach for the cheapest one that works:

• Deterministic assertions. Did the output parse as valid JSON? Does it contain the required field? Is the classification one of the allowed labels? Is the number within range? These are fast, free, and never flaky. Use them for everything you possibly can.
• Reference matching. For cases with a known answer — a category, an extracted value, a yes/no — compare against the label directly. Exact match, or fuzzy match with a threshold.
• LLM-as-judge. For open-ended outputs where there's no single right answer — a summary, an explanation, a tone — ask a model to grade the output against a rubric.

LLM-as-judge is powerful and genuinely necessary for subjective outputs, but treat it with care. The judge is itself a prompt, with its own failure modes. Give it a concrete rubric rather than "rate this 1-10." Ask it to grade specific dimensions — is it factually grounded in the input, does it answer the question, is the tone right — rather than a vague overall vibe. Pin the judge model and version so your scores don't silently shift when the provider updates the model. And spot-check the judge against human judgment periodically; a judge that has quietly drifted is worse than no judge, because it gives you false confidence.

Whatever the grader, the output of a run is a score: what percentage of cases passed. Now "better" has a number attached, and you can compare two versions of a prompt instead of arguing about two anecdotes.

Regression testing: the change that fixes one and breaks three

This is the failure mode that vibes-based tweaking is completely blind to, and it's the reason evals matter most.

You get a complaint: the assistant is too verbose in one situation. You add a line to the prompt — "be concise." You check that one case, it's better, you ship. What you didn't see is that "be concise" also stripped the reasoning out of three other cases where the detail was the whole point. You fixed one and broke three, and you won't find out until three more complaints trickle in over the next two weeks.

An eval harness catches this the moment you make the change. You run the full suite, and instead of one case looking better, you see the aggregate: case A went from fail to pass, but cases B and C went from pass to fail. Net negative. The change doesn't ship, or it ships alongside a fix that protects B and C. Either way, you knew before your users did.

This is exactly what a regression test suite does for ordinary code, and prompts need it more, not less, because their behaviour is diffuse. A code change has a defined blast radius you can reason about. A prompt change can shift behaviour on inputs that have nothing obviously in common. The suite is how you see the blast radius you can't reason about.

The prompt lifecycle

Put it together and a prompt has a lifecycle, the same way a feature does.

• Draft. Write or change the prompt in a branch. This is where the old craft still lives — wording, structure, examples all matter. But it's the start of the process, not the whole of it.
• Eval. Run the full suite. The change has to clear the bar: no net regression, and ideally a measurable gain on the cases it targets. This gate runs in CI on every pull request, so a prompt change that drops the score fails the build exactly like a broken test.
• Deploy. Ship it — ideally behind the same progressive rollout you'd use for code. A canary on a slice of traffic, watched, before it reaches everyone.
• Monitor. Watch how it behaves on real inputs, which are always stranger than your test set.
• Iterate. Feed what you learn in production back into the case set. Every real failure becomes a new labeled case, so the same failure can never ship twice.

That last arrow — production failures becoming new eval cases — is what makes the loop compound. Your suite gets stronger every time reality surprises you, and the surface area you can safely change grows over time instead of shrinking under fear.

Structured outputs as contracts

There's a second discipline that makes all of this tractable: make the output structured and validate it against a schema.

A prompt that returns free-form prose is hard to test and hard to consume. A prompt that returns JSON conforming to a schema is a function with a typed return value. You get:

• Cheap deterministic grading. Half your eval assertions become "does it match the schema," which is free.
• A contract with the calling code. The code downstream knows the shape it's getting, and schema validation at the boundary catches malformed output before it corrupts anything.
• A place to fail loudly. When the model returns something off-schema, you reject it and retry or fall back, rather than passing garbage downstream.

Use the model's native structured-output or tool-calling mode to constrain the shape, and still validate on your side — the constraint reduces schema violations but doesn't eliminate them, and the values inside a valid shape can still be wrong. Schema validation is the contract; the evals are what tell you the values inside that valid shape are actually good.

Observability in production

Evals tell you how a prompt does on the cases you've collected. Production tells you about the cases you haven't. You need to see both.

Log the real inputs, outputs, latency, token cost, and any structured signals — did the output parse, did validation pass, did the user retry or abandon, did a downstream step fail. Track these over time and per prompt version, so when you deploy a new version you can watch the parse-failure rate, the cost per call, and the retry rate move in real time. Sample the logged traffic and route interesting cases — failures, outliers, low-confidence outputs — back into the eval set. Production observability and the eval harness aren't two systems; they're two ends of the same loop, and the traffic flows from one into the other.

mistakes

A few that show up again and again:

• Evaluating on invented cases. The cases you make up are the ones the prompt already handles. Your suite has to be built from real traffic or it measures nothing.
• A too-small suite. Five cases give you a score that swings wildly on noise. You need enough cases that a one-point move means something.
• Trusting the LLM judge blindly. The judge is a prompt with its own failure modes. Give it a rubric, pin its version, and check it against humans, or it will confidently lie to you.
• Grading with a judge when an assertion would do. LLM-as-judge is slow and costs money. Every case you can grade with a deterministic assertion is a case you should.
• Prompt and parser drifting apart. When the prompt changes what it outputs but the code that parses it doesn't change with it, you get silent breakage. Keep them in the same commit.
• No production feedback loop. If real failures don't become new eval cases, the same failure ships twice. The loop has to close.
• Skipping the CI gate. An eval suite that only runs when someone remembers to run it is a suite that doesn't run. Wire it into the pull request check.

TL;DR

A production prompt is code. It needs the same discipline: live in version control, changed through reviewed pull requests, tested against a suite of real labeled cases, gated in CI so a regression fails the build, deployed behind a progressive rollout, and monitored in production with failures fed back into the case set.

The wording still matters — that's the draft step. But wording is where the process starts, not where it ends. Vibes-based tweaking fixes the case in front of you and breaks the ones you can't see. A tested pipeline tells you the blast radius before your users find it.

If your prompts are drifting without tests, let's talk.