Operations
Intermediate
16 min
Prompt version control and A/B testing
The problem
Someone improved the customer-reply prompt to make responses friendlier. Two weeks later the drafts are missing a policy caveat they used to include, and nobody can find the previous version. The friendliness change had removed a constraint that was doing real work, and there is no way back.
The workflow
1Identify the prompts that matter
2Version them
3Write the standard
4Change one thing
5Run 20 inputs both ways
6Score blind
7Adopt or revert
★Record why
What you need
Prompt's job: draft customer email replies, seen by customers. Current version: includes a constraint 'always state the return window that applies to this category, from the knowledge base'.
The tool
Your prompt library's version and notes fields
The prompt
Subscribers. The full prompt is part of the SMEMinds AI Playbook. See plans.
What comes back
1. THIS IS ONE CHANGE — good.
Verify before you use it
Human review required. Check every line below before this leaves your screen.
- Check that version B genuinely changed only the tone instruction. Verify the return-window constraint is character-for-character identical.
- Include the angry emails in the test set. A tone change looks best on routine inputs and worst where it matters.
- Set the decision rule before scoring. It is the only defence against your own preference for your change.
- Blind the scorer properly. Unblinded scoring of your own change is not a test.