--- title: "How to set up a Surprise & Delight A/B test" slug: "how-to-set-up-a-surprise-delight-ab-test" updated: 2026-10-29T21:07:27Z published: 2026-10-29T21:07:27Z canonical: "help.skio.com/how-to-set-up-a-surprise-delight-ab-test" --- > ## Documentation Index > Fetch the complete documentation index at: https://help.skio.com/llms.txt > Use this file to discover all available pages before exploring further. # How to set up a Surprise & Delight A/B test Surprise & Delight A/B testing lets you compare two versions of a rule's reward, such as a free gift versus a discount, to see which one keeps more subscribers. Each test also shows the cost per retained subscriber, so you can confirm your reward spend is actually paying off. ## Before you start - **Start from an existing rule:** Tests are created from a Surprise & Delight rule. Learn how to create one in the [Surprise & Delight guide](https://help.skio.com/docs/en/how-to-set-up-a-surprise-delight-rule). - **Every-order rules can't be tested:** The **Start A/B test** option isn't available on rules that trigger on every order. - **Turn off the older A/B setting:** If the rule has **A/B testing (optional)** turned on in its settings, turn it off before starting a test. - **One test per rule:** Each rule can have one open test at a time, and a draft counts as an open test. - **Only the reward changes:** Variant B uses the same trigger and conditions as the original rule. You can only change the reward. - **The rule locks during the test:** From the moment you save a draft until you end, cancel, or promote the test, you can't edit or delete the rule. Make any other changes you need before you start. ## What is Surprise & Delight A/B testing? An A/B test splits subscribers who qualify for a rule between two versions of its reward, then tracks which version retains more of them. - **Variant A (Original):** Your current rule, exactly as it's live today. - **Variant B (Modified):** A copy of your rule with a different reward, such as a different gift, quantity, or discount. Each subscription is assigned to one variant and keeps it for the length of the test, including on repeat milestones. If the rule has **Apply per customer** turned on, assignment is per customer instead, so all of a customer's subscriptions get the same variant. > [!TIP] > Keep Variant B as close to the original as possible. If you change the gift and the discount at the same time, a winning result won't tell you which change made the difference. ## How do I set up a Surprise & Delight A/B test? Start a test from a rule's menu, configure the split and stopping condition, edit Variant B's reward, then publish the test. ### Step 1: Navigate to Surprise & Delight In the left-hand menu of your **Skio Dashboard**, go to **Retain** > **Surprise & Delight**. Your rules are on the **Rules** tab, and your current and past tests are on the **A/B Testing** tab. ### Step 2: Start a test from an existing rule 1. On the **Rules** tab, find the rule you want to test. 2. Click the **⋮** menu on the rule card and select **Start A/B test**. ### Step 3: Configure your test The **Create new test** panel opens with your original rule selected. Fill in the following: - **Test name:** Required. A name that describes what you're testing. - **Test description:** Optional. Your hypothesis, such as `Testing whether a 10% discount outperforms a free sample at order 3`. This makes results easier to interpret later. - **Traffic split:** Use the slider to set the share of subscribers who get each variant, from 10% to 90%. It defaults to 50/50, which reaches a result fastest. - **Variant source:** **Duplicate current rule** clones your original rule so you can edit the copy as Variant B. - **Winning metric:** `Retention past next order`. Skio picks the variant that keeps more subscribers active through the order that includes their reward. - **Notify me to stop test:** Turn this on to get an email when your results are ready. - **Stopping condition:** Choose **Specific date** to stop the test on a set date, or **Data confidence** to stop it once results reach your **Statistical significance goal** (`90%`, `95%`, or `99%` confidence). Higher confidence needs more subscribers and takes longer. Confidence-based tests stop after 100 days even if the goal isn't reached. Click **Next** when you're done. ### Step 4: Edit Variant B's reward Clicking **Next** opens the test page with three cards. **Variant A · Original** shows your current reward, and **Shared by both variants** shows the trigger, conditions, and split. Both are locked and can't be changed in a test. 1. In the **Variant B · Modified** card, change the reward you want to test. Use **Change product** or **Quantity** to edit the gift, **Add discount** or **Add free gift** to add a reward, or **Enable gift swapping for customers** to let subscribers choose a different gift. 2. Click **Create test** in the top-right corner. Your test is saved as a draft, and nothing is live yet. ### Step 5: Publish your test 1. Open your draft test from the **A/B Testing** tab. 2. Click **Publish test**, then click **Publish test** again to confirm. Eligible subscribers start being split between both variants from this point on. ## How do I read my test results? Open the **A/B Testing** tab to see every test, its current confidence level, and how each variant is performing over time. The cards at the top summarize all of your tests that have been published: - **Subscribers in tests:** Total participants across your tests. A subscriber who reaches a repeat milestone is counted again. - **Reward spend:** Total cost of rewards on orders that have billed. - **Avg retention lift:** The difference in retention between Variant B and Variant A, in percentage points. Tests with more participants count more toward the average. Each test card shows: - **Confidence:** How confident Skio is that the difference between variants is real and not random. Each variant needs at least 200 participants before a winner can be flagged. - **Participants:** Subscribers assigned to either variant. - **Cost / retained sub:** Each variant's reward spend divided by the subscribers it retained, shown as Variant A / Variant B. Free gifts are counted at the product's price, and discounts count once an order bills with them. - **Retention chart:** Each variant's retention rate over the course of the test. A subscriber counts as retained if the order that includes their reward bills before they cancel. ## What do the test statuses mean? Each test card shows a status badge so you can tell at a glance where the test is. - **Draft:** The test is saved but not published. No subscribers are being split yet. - **Active:** The test is live and subscribers are being split between variants. - **Finished:** The test reached its stopping condition, or you've ended or promoted it. Once a test reaches its stopping condition, new subscribers stop joining, and subscribers already in the test keep their variant until you choose what to do. - **Winner · Variant A** or **Winner · Variant B:** Shown next to **Finished** when you promoted that variant. - **Cancelled:** The test was cancelled before it finished. ## How do I end a test? Skio checks your test daily and marks it `Finished` once it reaches its stopping condition, but it never changes your live rule on its own. The button on the test card depends on the result. ### Promote a winner If the result is statistically significant, the test card shows **Select winner**. 1. Click **Select winner** to open the **Conclusion & promotion** dialog. 2. Review the **Winner** and **Underperformed** labels and each variant's retention rate. 3. Select the variant you want to keep, then click **Promote Variant A** or **Promote Variant B**. You can promote either variant. The test stops immediately. Skio replaces your original rule with a new rule that uses the promoted variant's reward, and deletes the original rule. The test card shows a **Winner** badge and its end date. > [!WARNING] > If you'd rather keep your original rule without promoting, click **End without promoting** instead. This is final, and you can't promote a variant from that test afterward. ### End a test without a winner If the result isn't statistically significant, the test card shows **End test**. 1. Click **End test**. 2. Click **End test** again to confirm. Your original rule (Variant A) goes back to running for all subscribers. Any Variant B rewards on orders that haven't shipped yet are removed. The test's results stay on the **A/B Testing** tab. ### Cancel or change a test You can cancel a test at any point before it's resolved, including while it's a draft. Open the **⋮** menu on the test card and select **Cancel test**. To change a running test's stopping condition, select **Edit termination** from the same menu. ## FAQ --- ### FAQ #### Can I edit a rule while its A/B test is running? No. You can't edit or delete a rule while its test is a draft, active, or finished and waiting on your decision. To make changes, cancel or end the test first. You can still duplicate the rule. #### Can I pause a test? No. To stop a test, end it, cancel it, or promote a variant. #### Can I run more than one test at a time? Yes, as long as each test is on a different rule. If a subscriber qualifies for more than one test, Skio assigns them to only one. Rules that aren't part of a test keep applying as usual. #### What if my test has low participation? Each variant needs at least 200 participants before Skio can flag a winner. Rules that trigger rarely, like a reward on order 12, can take a long time to get there. Consider a lower significance goal, a longer test, or testing a rule that triggers earlier in the subscriber lifecycle. #### What's the difference between this and the A/B testing (optional) setting on a rule? **A/B testing (optional)** is an older setting inside a rule that gives the reward to a percentage of subscribers. It's separate from tests on the **A/B Testing** tab, and you need to turn it off on a rule before you can start a test. #### What if a free gift goes out of stock during a test? The test keeps running. Skio doesn't check stock levels for Surprise & Delight rewards, including during a test. #### Who gets the results email? If **Notify me to stop test** is on, Skio sends one email to your store's notification email when the test finishes. The email refers to Variant A as Control and Variant B as Treatment. No emails are sent when you promote, end, or cancel a test. #### Do subscribers know they're part of a test? No. Subscribers see their assigned reward the same way they'd see any Surprise & Delight reward, in the Customer Portal and through any notifications you've turned on.