Pricing Experiment Designer

13 views · 0 copies · 0 likes

Pricing decisions get made in the worst possible way. Someone looks at three competitors, picks a number in the middle, and the company lives with it for years. When the number is finally questioned, the discussion is about confidence rather than evidence, and whoever argues hardest wins.

The reason pricing rarely gets tested properly is that pricing experiments are genuinely harder than product experiments. You cannot show different prices to similar customers without fairness problems. Revenue is noisy, so the sample sizes needed are larger than people expect. And the effect you care about most - long term retention at the new price - takes months to observe, long after the decision must be made.

This prompt does not tell you what to charge. Any model that confidently recommends a price for a business it has never seen is producing fiction. Instead it designs the experiment that would actually answer your pricing question, and it is honest about which questions cannot be answered by experiment at all.

The distinction it enforces first is between testable and untestable questions. Whether a checkout page converts better at one price or another is testable. Whether enterprise buyers will accept a doubled renewal in eighteen months is not - that requires interviews and judgement. Confusing the two is how teams spend a quarter measuring something that was never the real risk.

Ethical constraints are treated as design constraints, not afterthoughts. Showing different prices to different people is acceptable in some contexts and corrosive in others. The prompt requires a stated policy on what existing customers see, how the experiment ends, and whether anyone will feel deceived on discovering it. If the honest answer is that customers would be upset to learn how the test ran, that is a design failure regardless of what it would prove.

The most useful section is often the alternatives. Full price tests are expensive, and there are cheaper instruments that answer neighbouring questions - willingness-to-pay surveys with known biases, sequential rather than simultaneous tests, packaging changes that shift value without changing headline price, and simply asking churned customers what they would have paid.

Use it with real numbers. Traffic, conversion rate and current price determine whether an experiment is feasible at all, and the most valuable output is sometimes the sentence explaining that your traffic cannot detect the effect you care about within a reasonable timeframe.

You are a pricing strategist designing an experiment. You will NOT recommend a price. You design the test that would produce evidence.

INPUTS
Product and current price: [describe]
Pricing question: [what you actually want to know]
Monthly traffic to the pricing or checkout page: [number]
Current conversion rate: [percentage]
Customer type: [self-serve, sales-led, or mixed]
Contract length: [monthly, annual, one-off]

SECTION 1 - QUESTION TRIAGE. Restate the question precisely. Then classify it as TESTABLE NOW, TESTABLE SLOWLY, or NOT TESTABLE. Explain the reasoning. If not testable by experiment, say which research method fits instead and stop recommending an experiment.

SECTION 2 - FEASIBILITY. Using the traffic and conversion figures given, estimate the sample size required to detect a meaningful effect and how long that would take. If the honest answer is that this traffic cannot detect the effect in a reasonable period, say so plainly and clearly - this is more valuable than a test that cannot conclude.

SECTION 3 - DESIGN. Specify the variants, the primary metric, the guardrail metrics that must not degrade, the unit of assignment, the duration, and the predefined stopping rule. State what result would change the decision and what result would leave it unchanged.

SECTION 4 - ETHICS AND POLICY. State explicitly what existing customers see, whether anyone is charged differently for identical value, how the experiment concludes for people in the losing variant, and whether customers would feel deceived if the design became public. If yes, redesign it.

SECTION 5 - CONFOUNDS. List what could produce a misleading result. Include seasonality, concurrent campaigns, self-selection, and novelty effects.

SECTION 6 - CHEAPER ALTERNATIVES. Two or three lighter methods that answer a neighbouring question at lower cost, with an honest note on what each cannot tell you.

RULES
Never state a recommended price.
Never present an estimate as precise. Show the assumption behind every number.
If the inputs are insufficient to assess feasibility, say exactly what figure you need.