The Coupon Worked. Did the Business?

There's a strange habit in ecommerce: brands will obsess over tiny changes to an ad, an email subject line or a product page then casually give away 20% of their revenue because “that’s what usually works". Ten percent feels a little weak. Fifteen percent feels respectable. Twenty percent feels like a real promotion. Free shipping is almost taken for granted.

Ask why, though, and the answers tend to get vague.

“We’ve always done 20%.”

“Our customers expect it.”

“Anything below 15% doesn’t move the needle.”

“Free shipping converts better.”

Maybe. But how do you actually know?

For a long time the discount strategy has been surprisingly difficult to test properly. A brand might run 20% off one weekend and compare the results with a normal weekend. Or it might compare this year’s Black Friday promotion with last year’s. Revenue goes up, conversion improves, everyone congratulates themselves and the discount gets added to next year’s promotional calendar.

The problem is that none of this tells you what the discount actually caused.

That distinction matters because Shopify is making it easier for merchants to treat promotions as experiments rather than articles of faith. With Rollouts merchants can coordinate changes to their storefront and commerce experience and, importantly, run experiments in which only part of eligible traffic receives a change. Discounts can now be part of that testing mindset. And that creates a much more interesting question than “Did our sale perform well?”

The question becomes: How much of that performance would have happened anyway?

Your biggest discount problem may be customers who didn’t need one

Imagine 100 people are about to buy from your store. Some are undecided. They like the product but the price is holding them back. Give them 20% off and they buy.

Great. The discount did its job.

But another group was going to buy regardless. They already wanted the product. They had accepted the price. They were halfway to the checkout when your promotional banner told them they could save 20%.

They buy too.

The discount code gets attached to both groups, so your analytics may make the promotion look successful in both cases. But economically those customers are completely different. The first group was persuaded. The second group was subsidized. You didn't create an additional sale with the second group. You simply collected less money for a sale you were probably going to get anyway. This is the uncomfortable part of discounting that ordinary promotional reporting rarely reveals. A dashboard can tell you that $200,000 of revenue used a promotional code. It cannot automatically tell you how much of that $200,000 was caused by the promotion.

That requires a control.

And once you have a meaningful control group, discounting stops being purely a marketing tactic and starts becoming something you can measure.

The old way of judging promotions is deeply flawed

Suppose you normally convert 3% of visitors. You launch a 20% discount and conversion jumps to 3.6%. At first glance that's a fantastic result. Conversion is up 20%. Someone takes a screenshot. It goes into the Monday meeting. The conclusion practically writes itself: 20% off works. Except you still don't know what would have happened without the promotion.

Maybe traffic quality improved that weekend. Maybe your paid media team changed its targeting. Maybe payday landed on Friday. Maybe an influencer mentioned one of your products. Maybe a competitor went out of stock. Maybe your email list was unusually engaged. Or perhaps demand was already rising. Comparing Saturday with the previous Saturday can be useful operationally but it isn't the same as running an experiment. A proper test asks two comparable groups of shoppers to behave under different conditions at roughly the same time. One group experiences business as usual. Another receives the offer. Now you're getting closer to answering the question you actually care about.

Did the discount change behavior?

That's a much higher standard than saying sales went up during a sale.

Shopify is bringing that question closer to the platform

This is what makes Rollouts interesting. At first glance Rollouts can sound like another release-management feature. Merchants can prepare changes and coordinate when they go live rather than manually switching pieces of the store at exactly the right moment. That alone is useful. Anyone who has managed a major ecommerce campaign knows how messy launches can become when a new theme, checkout changes, merchandising, promotions and other updates all need to appear together.

But experimentation changes the significance of the feature. Instead of exposing every eligible shopper to a change immediately, merchants can compare a treatment experience with a control experience. That means a promotional idea no longer has to begin as an all-store decision.

It can begin as a hypothesis. That's a subtle shift, but it could change the way ecommerce teams think about promotions. Instead of asking:

“Should we do 20% off?”

you can ask:

“Does 20% off generate enough additional demand to justify the margin we give away?”

Those are not the same question.

Conversion rate can lie to you

One of the easiest mistakes in discount testing is declaring whichever offer produces the highest conversion rate of the winner. Discounts are supposed to improve conversion. If you reduce the price enough you'd be surprised if purchasing didn't become easier. But the business isn't trying to maximize conversion at any cost.

Consider a deliberately simple example. You receive 100,000 visitors. At your normal prices 3% purchase and the average order value is $100. That's 3,000 orders and $300,000 in revenue. Now imagine a 20% discount pushes conversion to 3.5%. You get 3,500 orders instead.

Fantastic, right? Not necessarily.

If those baskets are now effectively generating around $80 instead of $100, 3,500 orders produce roughly $280,000 in revenue before we even consider changes in basket composition. Conversion went up. Orders went up. Revenue went down. And because those extra 500 orders also have costs attached to them - product cost, fulfillment, payment fees, packaging, customer service - the profit picture can be worse still.

This is why discount experiments need to be judged economically, not cosmetically. A higher conversion rate looks great on a dashboard. It does not pay the bills by itself.

The number worth watching is closer to profit per visitor

There isn't one perfect metric for every Shopify merchant. Margins, fulfillment costs, repeat behavior and product economics vary enormously. But the thinking should move toward contribution profit rather than conversion alone. Start with the revenue generated by a group of shoppers. Account for the discount. Subtract meaningful variable costs such as cost of goods, payment fees and fulfillment. Then relate what's left back to the amount of traffic that produced it.

In other words: how much economic value did each group of visitors actually create? Imagine your control converts at 3%, while the discounted treatment converts at 3.4%. Marketing sees the conversion lift and wants to roll the offer out immediately. But after accounting for the discount and variable costs the control produces $2.70 of contribution profit per visitor while the treatment produces $2.55.

You acquired more orders and created less value.

Now imagine another promotion barely changes conversion - 3% to 3.1% - but improves basket size enough that contribution profit reaches $2.90 per visitor.

Which promotion would you rather have?

This is where discount testing becomes more than a CRO exercise. Finance should care about these experiments just as much as marketing does. You're testing the price at which you sell your products. That's an economic decision.

Start with the uncomfortable test: discount versus no discount

Brands naturally want to test one promotion against another. Ten percent versus 20%. Twenty percent versus free shipping. $20 off versus 15%. Those tests can absolutely be useful. But they can also skip the biggest question.

What if you didn't offer the discount at all?

Imagine a brand that has offered 20% off during every major holiday for five years. Ask the team why and you'll probably hear something like, “Our customers expect 20%.”

Perhaps they do. But that's still a hypothesis.

What happens if a control group sees the normal store experience while another group receives 20% off? If the discounted group dramatically outperforms the control on incremental profit you have strong evidence that the promotion is earning its keep. If the difference is surprisingly small you've discovered something potentially much more valuable.

A large share of your customers may have been willing to buy without the incentive. That's not bad news. That's margin waiting to be recovered.

Then find the minimum effective discount

Once you've established that an incentive creates genuinely incremental demand the next question isn't necessarily: “Would an even bigger discount work better?”

It's often the opposite. How small can the incentive become before its effectiveness materially declines?

Suppose 20% off performs well. Test 15%. If 15% generates almost the same purchasing behavior while preserving another five points of revenue, that's a meaningful win. Then perhaps test 10%.

There may be a point at which customers respond to the existence of an offer more than to its exact size. A shopper who needs a little push may be persuaded by 15% just as effectively as 20%. Without testing you'll never know. You'll simply keep paying the extra five percentage points because nobody wants to be responsible for changing the promotion that “works.”

This is how habits become expensive.

Free shipping deserves the same scrutiny

Free shipping is another ecommerce belief that often escapes serious testing. Customers hate unexpected shipping charges. There's plenty of reason to believe shipping economics matter. But “free shipping works” still isn't specific enough to be useful.

• Does unconditional free shipping beat 10% off?
• Does free shipping above $75 outperform 15% off everything?
• What happens if the threshold is $100? Or $125?
• Does the threshold encourage customers to add another product to their basket?

A threshold promotion can behave very differently from a blanket percentage discount because it changes the customer's goal. Instead of simply making the existing basket cheaper it can encourage the shopper to build a larger basket to unlock the benefit. That's exactly the sort of thing worth testing rather than debating in a meeting.

Keep experiments simple when you want to learn

One temptation with a tool like Rollouts is to change everything at once. New homepage. New promotional banner. New product merchandising. New checkout messaging. New discount.

Then treatment beats control by 12%.

Success. Except now you don't know why.

Was it the discount? The creative? The merchandising? The checkout change? Some interaction between them? There are times when testing the complete campaign experience makes sense. If the question is simply whether Campaign A beats the existing store test the package.

But if you're trying to learn something reusable about discount strategy, simpler experiments are usually more valuable. Change the thing you're trying to understand. Hold as much else constant as practical. That's less exciting than launching five clever ideas simultaneously but it gives you knowledge you can use again.

Don't kill the test on Tuesday morning

Another predictable problem will be impatience. The treatment jumps ahead during the first day and somebody declares victory. Or it falls behind and the team wants to shut it down. Real customer behavior is noisy.

Your weekday and weekend customers may behave differently. Traffic sources change. Returning customers may respond differently from first-time shoppers. Payday can matter. Email sends can temporarily distort the audience. Product availability changes. A handful of early orders can make a weak promotion look extraordinary.

Before launching an experiment decide what improvement would actually matter to the business. Decide which metric determines the winner. Decide how much evidence you need before you're willing to make the decision. Do that before seeing the results. Otherwise you're not really testing. You're watching numbers until they tell you what you hoped to hear.

Discounts can also change future behavior

There is one more problem that a short-term A/B test can't completely solve. Customers learn. If your store offers 20% off every three weeks eventually shoppers notice. Full price starts to feel less like the real price and more like the price impatient people pay.

Why buy today when another coupon is probably coming?

That effect is particularly dangerous for brands trying to maintain a premium position. A promotion can produce a beautiful seven-day conversion lift while slowly teaching customers that the product isn't worth its listed price. So the immediate experiment should only be the beginning.

Look at what discounted customers do later.

• Do they return?
• How quickly?
• Do they ever purchase at full price?
• Do they require another promotion before buying again?
• Did the campaign create new demand, or did it simply pull purchases forward from next month?

A discount can win the week and lose the customer relationship.

This changes the job of the ecommerce operator

The most interesting part of Shopify's move isn't the ability to test one more thing. It's what happens when experimentation becomes a normal part of running the store. Ecommerce teams already accept testing almost everywhere else. They test Meta ads. They test Google ads. They test email subject lines. They test landing pages, product imagery, calls to action and checkout flows.

Then the promotional calendar arrives and somebody says: “Let's do 20%.”

That decision can be more expensive than every button-color experiment the company runs all year. If you spend $50,000 on advertising people want to know what the money produced. If you unnecessarily give away $50,000 through discounting it can disappear into the vague category of “promotion.”

It shouldn't. Every discount is a bet.

You're betting that the additional demand created by the offer is worth more than the margin surrendered to provide it. Sometimes that bet will be excellent. Sometimes 20% off really will unlock enough hesitant customers to create significant incremental profit. Sometimes free shipping will transform the economics of a campaign. Sometimes a threshold offer will lift basket size enough to pay for itself several times over. And sometimes you'll discover that a large number of customers were going to buy anyway.

That's the discovery brands should be excited about. Because the purpose of experimentation isn't to prove that discounts work. It's to find out when they work, how much incentive is actually necessary, which customers need that incentive and whether the resulting demand is profitable.

So the next time someone in the meeting says: “We need 20% off. Ten percent won't work” don't spend 45 minutes arguing about it. Write down the hypothesis. Choose the metric that reflects the economics of the business. Protect a control group. Run the experiment. Then let your customers answer the question.

Stop asking whether 20% off works.

Test it.