A/B testing
Split your traffic between two versions of a widget — or two whole carts — and let the numbers pick the winner.
An experiment splits your storefront traffic between two versions and measures which converts better. Cartie assigns each visitor once and keeps them on the same version for the whole visit, so the comparison stays honest.
#Two kinds of test
| Type | What it compares | Use it when |
|---|---|---|
| Widget test | Two configurations of one widget in the live cart — different headline, threshold, colour, layout. | You want to know why something changed. One variable, one clean answer. |
| Cart test | Two complete carts against each other. | You have redesigned the cart, or want to try one of the templates against what you run today. |
#Creating a test
- Go to Experiments → New experiment
Choose whether you are testing a widget or a whole cart.
- Set up the variants
One variant is the control — what you run today. The other is the challenger. Give each a clear label; "Threshold $75" beats "Variant B" in three weeks' time.
- Choose the traffic split
50/50 is the default and reaches a decision fastest. You can go as lopsided as 90/10 to limit exposure to a risky idea, at the cost of a much longer test.
- Write the hypothesis
Optional but worth 20 seconds: "Raising the free-shipping threshold to $75 will lift AOV without hurting conversion." It is what makes the result actionable later.
- Start it
Experiments begin as a draft and do nothing until you click Start. Starting republishes your storefront config, so both versions go live immediately.
#Lifecycle
| Status | Meaning |
|---|---|
| Draft | Set up but not serving. Nothing has changed on your storefront. |
| Running | Traffic is being split and data is collecting. |
| Paused | Everyone gets the control again. Data already collected is kept, and you can resume. |
| Completed | Finished for good. The result is kept for reference; the cart keeps serving whatever is published. |
#The four decision gates
Cartie will not call a winner on a hunch. All four of these have to pass before a result counts:
| Gate | Threshold | Why |
|---|---|---|
| Sample size | 100+ conversions per variant | Below that, ordinary week-to-week noise looks like a result. |
| Runtime | At least 7 days | Shopping behaviour differs by day of week. A test that skips a weekend is measuring the weekday. |
| Statistical significance | 95% confidence or better | A two-proportion z-test against the control — the standard bar for "probably not chance". |
| Practical significance | Positive lift | A statistically certain 0.1% improvement is not worth shipping. |
#Reading the results
- Visitors — people who saw that version.
- Conversions — visitors who went on to place an order, attributed from Shopify's order webhook.
- CVR — conversions ÷ visitors for that variant.
- CVR lift — the challenger's conversion rate relative to the control.
- Confidence — how sure we are the difference isn't chance.
- Revenue — order revenue attributed to that variant.
#After a winner
Conclude the experiment, then make the winning configuration your live one — edit the widget in the builder and publish, or publish the winning cart from My Carts. Then start the next test: the compounding comes from running them continuously, not from one big win.
#Test ideas that usually pay
- Your free-shipping threshold — the most-tested number in ecommerce for good reason.
- Upsell row above the items vs. below them.
- Express wallet buttons above vs. below the product list.
- A rewards ladder in place of a single free-shipping bar.
- Adding urgency (countdown, low-stock) vs. a calm cart — it does not win for every brand.
Still stuck on this?Send us your store domain and what you’re seeing — we’ll take a look.