A/B testing button labels

Starting a test

Open EM Buttons → Reports and find Button Label Experiments. Choose a marketplace, type the alternative label you want to test, and start the experiment. Visitors are then split evenly between your current label and the new one.

One experiment runs per marketplace at a time. Other marketplaces are unaffected and keep their normal labels.

How the split works

A visitor is assigned to a variant in their browser and the choice is stored in a cookie holding a single letter. Nothing about the visitor is recorded. Because the assignment happens in the browser, a full-page cache can serve identical HTML to everyone and the split still works.

Reading the result

The experiment card shows views, clicks and click rate for each variant, with a badge summarizing the position. The badge holds back a verdict until each variant has at least 100 views and the experiment has at least 10 clicks in total, so an early run of luck does not read as a result.

Once there is enough data the plugin runs a two-proportion z-test and reports a winner at 95% confidence. Until that threshold is met it reports which variant is ahead and says the difference is not yet significant.

Finishing

Two buttons end an experiment. Use New Label and Finish makes the tested label the marketplace’s permanent button text. Keep Current Label and Finish leaves the label alone. Either way the result is stored under past experiments, with its numbers, and tracking rows for that experiment are cleared.