Operations Intermediate

Design a fair A/B test for a prompt change

Someone improved the customer-reply prompt to make responses friendlier.

Subscribers. The full prompt is part of the SMEMinds AI Playbook. See plans.

Before you use the output

Human review required. AI output is a confident first draft, not a verified answer.

Check that version B genuinely changed only the tone instruction. Verify the return-window constraint is character-for-character identical.
Include the angry emails in the test set. A tone change looks best on routine inputs and worst where it matters.
Set the decision rule before scoring. It is the only defence against your own preference for your change.
Blind the scorer properly. Unblinded scoring of your own change is not a test.