Best practices for Smart Upsell A/B testing

Before you start

This article assumes you know how to create and launch a test. If you haven't set one up yet, start with How to set up a Smart Upsell A/B test.

Why setup choices matter

A Smart Upsell A/B test only tells you something useful if the result is clear. The change you test, the winning metric you pick, and how long you let the test run all decide whether you end up with a confident answer you can act on, or a result you can't explain.

Who this is for

Merchants with an active Smart Upsell carousel who want to know whether a change will grow upsell revenue before rolling it out to every subscriber.
Merchants with segmented rules, such as loyalty tiers or tagged customers, who want to compare what works for different audiences.

Set up your test for a clear result

A few settings make the biggest difference to whether your test reaches a confident result.

Change one thing: Make a single edit to Variant B and keep the Change meter at 1 change. If you change several things and Variant B wins, you won't know which change caused it.
Match the metric to your goal: Choose Revenue per visitor when you care about upsell revenue, which is true for most tests. Choose Attach rate when you're testing whether customers engage with the carousel at all, such as with a new title or badge.
Keep the split even: A 50/50 split reaches a result fastest. Only shift it if you want to limit how many customers see an untested change.
Start with 95% confidence: It's a good balance of reliability and speed. Use 90% on a lower-traffic store, or 99% when the change is high stakes and you have plenty of traffic.
Give it enough time: Set Stop after at most long enough to cover at least one full billing cycle, since upsell revenue isn't recorded until the customer's next order bills.

Common use cases

Each use case below changes one thing in Variant B, so the result points to that change.

Testing curated products against best sellers

Scenario: Your top rule shows Best selling products, but you think a hand-picked set of complementary products would sell better.

Recommended setup: In Variant B, change only that rule's Promotion Group Type to Selected products and add your curated picks. Use Revenue per visitor as the winning metric.

Why this works: Different product sets often have different prices. Revenue per visitor accounts for both how often customers add something and how much it's worth, so a cheaper set that gets more adds doesn't win by default.

Tip: Make sure every product you add is enabled for one-time upsell, or it won't appear in the carousel.

Testing badge copy or color

Scenario: You show badges on your upsell products and want to know if different wording or a more visible color gets more customers to add something.

Recommended setup: In Variant B, change only the Badge text or the badge colors on one rule, not both. Use Attach rate as the winning metric.

Why this works: A badge affects whether customers notice and act on a product, not what the product costs. Attach rate measures that behavior directly.

Tip: Test the rule your largest audience sees, such as an Everyone rule, so the test collects data faster.

Testing rule priority

Scenario: You have a rule for a specific audience, like a loyalty tier, sitting below your general best sellers rule, and you want to know if showing it first performs better.

Recommended setup: In Variant B, move the targeted rule up one priority and leave everything else the same. Use Revenue per visitor as the winning metric.

Why this works: Rule order decides which products customers see first. Changing only the order isolates whether placement, rather than the products themselves, drives more upsell revenue.

Tip: Only customers who match the targeted rule see a difference, so this test can take longer to reach a result. Consider a longer Stop after at most duration.

Testing the carousel title

Scenario: Your carousel uses the default You might also like title, and you want to try copy that fits your brand or gives customers a reason to add something.

Recommended setup: In Variant B, change only the Carousel title under Global Settings. Use Attach rate as the winning metric.

Why this works: The title applies to every rule, so every customer in Variant B sees the change. That makes it one of the fastest tests to reach a result.

Common mistakes to avoid

Stacking several changes into one test: If Variant B changes the products, targeting, and title at once, a win won't tell you which change worked. Run each idea as its own test, one after another.
Ending a test early because it looks like it's winning: End test now abandons the test, and you can't promote either variant from it. Let the test reach its stop rule, then use Select winner.
Judging revenue in the first few days: Upsell revenue is recorded when the customer's next order bills, so revenue can look low early in a test even when customers are adding products.
Using a high confidence goal on a low-traffic store: A 99% goal needs a lot of traffic. On a smaller store, the test may hit its maximum duration before reaching a result. Try 90% or 95% instead.
Turning off the carousel or its products mid-test: If Smart Upsell is switched off, all rules are disabled, or a rule's promotion group no longer exists, neither variant gets views and the test can't produce a result.