How to set up a Smart Upsell A/B test

Before you start

Smart Upsell must be Active with at least one rule set up. Learn how in the Smart Upsell guide.
Your carousel needs enough traffic to produce a meaningful result. If traffic is low, Skio may block the test or warn you that it could take a long time to reach a result. Revenue per visitor needs more traffic than Attach rate.
You can have one Smart Upsell A/B test at a time, and a draft counts as your one test. Finish or delete your current test before starting a new one.
While a test is a draft, your live configuration stays editable. Once you launch, both variants are frozen until the test ends. Only the stop rule can be changed while a test is running.
Only views and add-ons from the upsell carousel in the Customer Portal count toward results. Products added to a subscription from the Skio Dashboard or elsewhere don't count.
Skio never promotes a variant automatically. You choose what happens when a test ends.

What is Smart Upsell A/B testing?

Smart Upsell A/B testing splits your customers between your current upsell configuration and one modified version, then shows you which one performs better. You can test almost any part of your carousel, including the carousel title, badges, audience targeting, rule priority, and upsell products.

Variant A (Original): Your current Global Settings and rules, exactly as they're live today.
Variant B (Modified): A copy of your current configuration that you edit with the change you want to test.

Each customer is assigned to one variant and sees that same variant on every visit for the length of the test.

Results reflect your whole upsell configuration, not just one change. For the clearest read, keep Variant B as close to the Original as possible and test one change at a time.

How to set up a Smart Upsell A/B test

Step 1: Navigate to the A/B testing tab

In the left-hand menu of your Skio Dashboard, go to Grow > Smart Upsell.
Click the A/B testing tab, next to Smart Upsell, to see your current and past tests.

Step 2: Configure your test

Click Start A/B test to open the Start an A/B test panel, then fill in the following:

Test name: A name that describes what you're testing (for example, Bestsellers vs curated picks).
Hypothesis: Optional. What you expect to happen, so the result is easier to interpret later.
Variant source: Duplicate current configuration is selected by default. Both variants start identical, so every change you make to Variant B is a change from what's live today.
Traffic split: The share of customers who see each variant. Each variant gets between 10% and 90% of customers. An even 50/50 split reaches a result fastest.
Winning metric: The metric Skio uses to pick a winner. You can't change it once the test is running.
- Revenue per visitor (recommended): Upsell revenue divided by everyone who saw the carousel.
- Attach rate: The share of customers who saw the carousel and added something from it.
Stop the test when: Choose how the test ends.
- The result is confident enough: Skio checks daily and stops the test as soon as the difference between variants reaches your Confidence goal (90%, 95%, or 99% confidence). Higher confidence needs more traffic and takes longer.
- A date I choose: The result is calculated once, on the date you pick.
Stop after at most: The maximum number of days the test can run, from 7 to 90 days (30 by default). This ceiling applies to either stop rule, so a test that never reaches its confidence goal still ends.
Email me when the test concludes: Get an email when results are ready. If you chose a date stop, you also get a reminder the day before.

Click Create draft to save your test.

Step 3: Edit Variant B

After you create the draft, Skio opens the test editor. Variant A (Original) is shown on the left and locked. Variant B (Modified) is on the right, and everything you change there is what Variant B customers see.

In Variant B, you can change:

Global Settings: The Carousel title and whether to Show badges.
Rules: Each rule's Promotion Group Type, Audience Targeting, Number of promoted products, upsell products, and badge text and colors. You can also rename, reorder, and add rules with + Add rule.

Click Save draft to save your progress, or Reset to original to undo all changes to Variant B.

Use the change meter

The Change meter at the top of the editor counts how many edits you've made to Variant B. One change gives you the cleanest data, because any difference in results can be traced back to it.

No changes: Both variants are identical, so the test can only measure the split itself.
1 change: Best for clean data.
More than one change: Skio shows a You changed more than one thing warning listing what you changed. If Variant B wins, you won't know which change caused it.

Best practice: Test one change at a time. To test several ideas, run them as separate tests one after another.

Step 4: Launch your test

When Variant B is ready, click Launch test. The test moves from Draft to Running, customers start being assigned to a variant, and both variants are locked until the test ends.

Managing a running test

You can pause, adjust, or end a running test from its card on the A/B testing tab. Click the ⋮ menu on the test card and choose an option:

Pause test: Temporarily stop the split. While a test is paused, every customer sees your live configuration.
Edit stop rule: Change the stop rule, confidence goal, maximum duration, or email setting. Customers stay on the variant they're already assigned to. The winning metric can't be changed. To use a different metric, end this test and start a new one.
End test now: Abandon the test without a result. See Ending a test early.

Viewing test results

The A/B testing tab shows a summary of all your tests, followed by a card for each test. Use the All, Active, Paused, and Finished filters to narrow the list.

Test impressions: Customers who saw a carousel while a test was running.
Avg. lift: The average lift across tests that reached a result.
Revenue attributed: Upsell revenue recorded against a test variant.

Each test card shows its status, goal and stop rule (for example, Ends at 95% confidence), Lift, Confidence, Customers, and Revenue, plus a trend chart once the test has been evaluated on more than one day. When there's enough data, a banner tells you which variant is winning, by how much, and at what confidence.

Test detail view

Click View test on a test card to see both configurations side by side, with each variant's traffic split, Global settings, and rules in priority order. Click Expand all to open every rule.

Skio tags anything in Variant B that differs from the Original:

Modified: A rule whose settings changed. The specific changed field, such as Badge colour, is tagged inside the rule.
Moved: A rule whose priority changed.

The Shared by both variants section shows settings that apply to both sides of the test, such as how customers are assigned and whether out-of-stock products are shown.

The Metrics section shows a trend chart you can view By day or By week, and a table comparing both variants on Attach rate, Revenue per visitor, Customers reached, and Customers who added.

Upsell revenue is recorded when the customer's next order bills, and appears on the test a few hours after that. Revenue can lag behind attach rate early in a test.

How test metrics are calculated

All metrics are calculated per variant, based on customers who saw that variant's carousel in the Customer Portal.

Revenue per visitor: Upsell revenue divided by everyone who saw the carousel. It only counts revenue from products added through the carousel, not the rest of the order.
Attach rate: The share of customers who saw the carousel and added something from it.
Customers reached: Distinct customers who saw the carousel.
Customers who added: Of the customers reached, the ones who added a product.

Ending a test and promoting a winner

When a test reaches its stopping condition, Skio marks the results ready for review but doesn't change your live configuration. If you turned on Email me when the test concludes, you'll get an email.

Promoting a winner

Click Select winner to open the Conclude this test dialog, then choose what to do:

Promote and stop: Select the variant you want to keep, then click Promote and stop. That variant becomes your live Smart Upsell configuration. You can promote either variant, even if it didn't win.
Stop without promoting: Leave both variants unselected, then click Stop without promoting. Your Original stays live and the test shows as Ended without promoting. You can also do this from the ⋮ menu on the test card.

Either way, the test's results stay available to review.

Ending a test early

To stop a running test before it reaches its stop rule, click the ⋮ menu on the test card, select End test now, then click End test to confirm.

The split stops immediately and every customer goes back to your live configuration. The test is marked Cancelled, you can't promote either variant from it, and your configuration unlocks for editing.

Ending a test early can't be undone.

If neither variant's carousel was viewed while the test ran, the test ends without a result and shows a No views were recorded notice.

FAQ

Can I edit my Smart Upsell rules while a test is live?

No. Both variants are frozen once a test launches. The only settings you can change are the stop rule settings, using Edit stop rule. To change anything else, end the test with End test now. You can keep editing your live configuration while the test is still a draft.

Why does my running test say nothing is being shown?

Neither variant has recorded a view yet. This usually means Smart Upsell is turned off, there are no enabled rules, or a rule uses a promotion group that no longer exists. The test can't produce a result until this is fixed.

Why does my test say no views were recorded?

Neither variant's upsell carousel was viewed in the Customer Portal while the test ran. Products added to a subscription from the Skio Dashboard or anywhere other than the carousel don't count toward test results.

Why is my test showing $0 revenue when customers have added products?

Upsell revenue is recorded when the customer's next order bills, not when they add the product. It then takes a few hours to appear on the test.

Does audience targeting still apply during a test?

Yes. Customers are split between variants, and each variant's rules still decide which products a customer sees based on their audience targeting.

Why is Start A/B test unavailable?

The button is unavailable if Smart Upsell isn't Active, if you already have a test (including a draft), or if your carousel doesn't get enough traffic to support a test. Finish or delete your current test to start a new one.