Articles

Guides
Resources

Split-Testing: How to Run a Revenue-Driven Test

Webinars

in Shopify
Events

by Joshua Uebergang
Tools Jul 21, 2016 10 minute read

Shopify Tutorials

Design Forums

About

Split-testing makes you a better designer because its the ultimate litmus test of revenue, handling client disputes, and satisfying
clients (you cant argue with a statistically proven 12 per cent increase in revenue).

And while tests can be done on micro-conversions, like form engagements or click-throughs, these types of actions are not
always correlated with revenue.

I could claim to oer visitors a free Ferrari if they visited the cart page. A lot more people would get one-step closer to the
purchase, but fewer would convert due to the distraction, deception, and ultimate disappointment.

Micro-conversions should be tracked but optimizations need to be revenue-driven. Prot is the essence of ecommerce, and few
retailers are optimized enough to warrant attention on micro-conversions.

But how do you actually perform a revenue-driven split-test in Shopify?

We know conversion optimization is key, and can see how split-testing near-instantly turns around an unprotable store. Yet it
isnt entirely clear to many designers and agencies how to A/B test a new product page, or how to test an entire theme.

Yet it can all be done.

Today, Ill show you, step-by-step, how I test anything in Shopify stores to inject prot into a clients bottom-line.

You might also like: How to Successfully Execute an Iterative Redesign in Shopify with Optimizely

The challenge with testing in Shopify

It is technically simple to run split-tests of visual elements on Shopify Plus stores. If you want to test Add To Bag text on the
button of your product page against Add To Cart, you create a jQuery that selects a unique CSS identier inside your favorite
split-testing tool. There is little dierence to testing within Shopify Plus compared to most websites because you have full control
over the checkout funnel.

The challenge with testing in Shopify is with the majority of merchants who use the checkout.shopify.com domain, and dont
have the required control over the checkout funnel. This means it can be hard (or impossible in some tools) to test for revenue.

It all has to do with cross-domain tracking with little munchies called cookies. Testing tools set a rst-party cookie, associated
with the stores domain. Due to browser restrictions, these cookies do not exist outside the domain. Once a visitor converts on
the Shopify checkout page, tools cannot link them back to the experiment.

As weve covered, revenue is critical to track. Safari is the primary browser known to block third-party cookies by default.
Browsers that do not automatically block third-party cookies, allow users the option to block them. Visitors are tracked on your
domain, but their conversions outside of your domain are lost on browsers that do not support these cookies.

This means you have to exclude a portion of users from the experiment right o the bat (namely Safari users), which can
introduce a bias to your results. If you view the Browser & OS report in Google Analytics, most stores will have browsers with
dierent conversion performance:

If you do not exclude these browsers from the experiment, your results can be skewed by those users who entered the test and
converted, but were not tracked because the third-party cookies were not available at the time of their conversion. Therefore, the
conversion rate reected in your results would not be accurate.

Even if browsers we know block third-party cookies and are excluded from the experiment, visitors using browsers like Chrome
and Firefox ,or ad blockers, may block third-party cookies in their settings. Since we have no way to determine this, the
conversions from users with this setting enabled, will not be tracked even though they were initially included in the experiment.

The best we can do is run a large number of conversions per variation (300 minimum) and integrate the data with analytics,
(which we will cover) so a breakdown can be done.

Finally, a smart store will have an abandoned campaign that markets to well-qualied visitors. Abandoned cart emails present an
unavoidable factor you should be aware of, which may not convert in the split-test. You will get sessions but no tracked
conversions.

The primary reasons for this are multi-device swapping (someone enters the test, abandons cart, then clicks the abandoned
email on another device and buys) or email campaigns that take visitors to resume orders at checkout.shopify.com (stores with
applications like Klaviyo do this).

Step-by-step on creating a split-test

The rst step is to identify what to optimize. This is a balance between what is easiest and what you estimate will create the
greatest prot.

Sometimes the best path to 50 per cent jumps in revenue is a whole new theme, which can easily take a month of programming.
You need to consider the cost of development, and the lost sales each day the site is unoptimized so the biggest prot
opportunity is not always the best.

I have two default recommendations. The rst is to make the simplest change that is likely to grow prot the most (low hanging
fruit). My second recommendation is to work closest to the payment as possible, then progress backwards along elements in
which every customer must engage. Tommy Walker discusses optimizing closest to the money in his article on reducing
shopping cart abandonment. The bigger the prot boost and the sooner it happens, the happier everyone will be.
An example of a simple change from Bryan Eisenbergs hierarchy of optimization is functionality.

Does the store satisfy a visitors needs? Every interested buyer wants to know how much their order will cost. If they cannot nd
clear costs for delivery, expect this change to increase prot. A lot of stores like BlackMilk have good success in mentioning their
delivery information in the header.

Here are a some of my favorite questions, taken from my book Shopify Conversion Rate Optimization, that you can ask in
surveys to solve problems at various levels of the hierarchy of optimization:

1. What almost stopped you from buying from us? The best question a merchant can ask on their thank you page.

2. What other options were you considering before buying from us? Another good question post-purchase to understand the
psychology of options and competition.

3. It seems you didn't complete your purchase, can you share why not? Great for when people leave the cart page or stay on it
for more than 20 seconds.

4. What other information would you nd helpful on this page? Trigger after 20 seconds to show only to an engaged visitor.
Good for product pages to ll in information gaps.

HotJar also has a great action plan and set of tools to help you identify what to test.

How to congure a test for Shopify

Youve learned about cookies and revenue tracking, now its important to select a testing tool that works in Shopify. The tool also
has to be aordable and simple to implement.

I also recommend picking an option that is fast, because we know from study after study that speed is correlated with revenue in
ecommerce. ConversionXL recently published a great study on the speed of split-testing tools:
When all these criteria are considered, your best options are Convert Experiments and Visual Website Optimizer (VWO). Other
tools, like Optimizely, are viable for Shopify Plus stores.

My preferred tool of choice is Convert Experiments. Each of the three tools I recommend have clear tutorials to set-up in Shopify:

Convert with Shopify

VWO with Shopify

Optimizely with Shopify

Once you have added the required scripts, congured the test settings, and set-up your goals, theres another critical integration
you must set-up in order to optimize for revenue.

Google Analytics integration

One last conguration you need before creating the actual split-test, is to integrate your testing tool with Google Analytics.
Always integrate with Google Analytics because of segmentation.

I had one test for a client that, although it increased total sales by 2.2 per cent, tablet sales dropped 20 per cent:

The sample data for tablet devices is one quarter of other devices, so that would play an inuence in the datas accuracy. Upon
further investigation through segmentation of tablet devices, Apple iPads had the 20 per cent drop in sales (again, insignicant
sample set). But this lead to a discovery of a 340 per cent drop in revenue for the tested variation on Apple iPads:

It turns out the test for an upsell/cross-sell used a media query for Apple iPads, which ruined the appearance of additional
product purchases for the order, and hence average order value.

The rst requirement of any analytics integration is to get a clean analytics set-up. Theres a lot more you need to do than setup
enhanced ecommerce tracking, like keep out spam hits, track your big customers across multiple devices, see the full funnel
process, set-up content groups, and use cross-domain tracking. Follow my Google Analytics Setup for Shopify to get clean data
in all these scenarios.

Most testing tools provide a good guide to integrate with Universal Analytics. Its been a whole year since I encountered a store,
let alone a website, on classic analytics. If you still use classic, update the code using the tutorial above.

Here are the main testing tools, and their steps to integrating with UA:

Convert integration with Universal Analytics

VWO integration with Universal Analytics

Optimizely integration with Universal Analytics

You might also like: A Short Tutorial for Running A/B Tests in Shopify with Optimizely

Video example of creating a split-test

Heres a short video tutorial I created that demonstrates an actual test being created for a Shopify store. This shows a simple
split-test, using the visual editor of Convert Experiments:

How to Create a Split Test in Shopify

Bonus testing tips for Shopify

How can I test price?

How can I test price?
You cant test price in Shopify with a split-testing tool because the tool alters data at the DOM level. Previously in Shopify, to test
a product's price you had to make changes to the product itself over a set period, then contrast it to the old price. It's imperfect
given the variable nature of trac sources and seasonal trends.

Now you can use the Quant Price app. Products with high turnover are the most suited to be tested. If you havent discussed
optimizing the pricing of products with clients on Shopify Plus, do it now. Its devastatingly eective the simplest thing an
established store can do to jack up prot.

Im having trouble designing my test. Any advice?

Its possibly a wiser step is to make the change on the site so it becomes the control. You can then create a test in the visual
editor of the testing tool, to bring it back to the original version. The amount of jQuery required can otherwise get convoluted for
complex tests.

How can I test alternate templates?

A new template can be created with the syntax:

default_template_name.*.liquid

If you append ?view=list to any collection URL, your store will use the collection.list.liquid template. If you use ?view=grid at
the end, the store will call the collection.grid.liquid template. Keir Whitaker has written about this method. The same goes
for other pages like the cart.

This is a less than ideal way to test a Liquid template on a page-by-page basis. Unfortunately, site-wide theme tests do not work
with this, because the view query string is lost with user navigation. The above is a better solution.

How can I test a new design site-wide?

This is a neat hack I gured out to test diverse design changes. You can test new product page designs, and have it stick across
all product pages for the duration of the test!

Duplicate the existing theme, then make changes as needed in the duplicate theme. Also add the following CSS:

.shopify-preview-bar { display: none }

The line of code removes the preview text at the bottom insert by Shopify, so users in the variation do not see it.

Preview the new theme from within the Shopify admin, then view the source code in the head to get the themes ID. In the below
example, we see the ID is 65568381:

<script>
//<![CDATA[
var Shopify = Shopify || {};
Shopify.shop = "mygoods.myshopify.com";
Shopify.theme = {"name":"[LIVE] My Goods","id":65568381,"theme_store_id":null,"role":"main"};
//]]>
</script>

Then, what we need to do is append ?preview_theme_id=65568381 as a variation for a URL test so that the new design loads for
50 per cent of people who visit the store. Here is exactly how to do it inside Convert Experiments:

Create an A/B URL test. Make the control URL:

http://www.store.com([^\?]*)??(.*)$

Use the following as the variation 1 URL, where the preview_theme_id variable is what you discovered from the source code.
This code appends the preview_theme_id variable to load a new theme:

http://www.store.com$1?preview_theme_id=65568381&$2
Then enable Support Regular Expressions. Your test variations will look like:

The last thing you need to do is include all pages in the test with the site area settings:

http://www.store.com/{0,1}(([\?&]{1}[^=]{1,}=[^=]*[&]{0,1})*)$

And exclude pages when

the query string contains:

preview_theme_id=65568381

Be sure there is the

necessary Convert Experiments testing code in the Liquid les of the duplicated theme. You can now test any design change you
want in Shopify.

Get testing
Hopefully, this tutorial and video have helped demystify some of the confusion around revenue-driven split-testing in a Shopify
store.

As we all know, prot is the essence of ecommerce, so make better design decisions today by testing for revenue!

You might also like: How A/B Testing Will Make You a Better Web Designer
About the Author
Joshua Uebergang is author of Shopify Conversion Rate Optimization, which is a free ebook that covers everything you need to
increase prot in Shopify. He runs Digital Darts to help Shopify stores grow. Hes also tall, and hates it when you ask how tall he
is within seconds of conversation.

Follow @digital_darts

TOPICS:
Resources Shopify Tutorials Web Design

Lets grow your freelance or agency business.

Get design inspiration, practical takeaways and free resources to help you nd and delight clients.

Email Address

Get updates

No charge. Unsubscribe anytime.

 d Welcome to Disqus! Discover more great discussions just like this one. We're a lot more than comments. Get Started

Dismiss

2 Comments Shopify Partner Blog Michael Speight

Sort by Best
Recommend Share

Join the discussion

pacman302 2 months ago

I tried "How can I test a new design site-wide?" following the exact steps in Convert but something is wrong with the url, when I
view my alternate design, no matter what I do, I get redirect to the home page. Any idea why?
Reply Share

Darwin Crew 5 months ago

Very useful list of web development blogs. It will be very helpful for
the developer and designer. Thanks for sharing. Keep it up..
Reply Share

ALSO ON SHOPIFY PARTNER BLOG

Everything You Need to Prepare for Black Friday and 15 Web Designer and Development YouTube Channels To
Cyber Monday Subscribe To
2 comments 2 months ago 1 comment 2 months ago
Eldar Galiev The holiday shopping season is the best time for Cam Great stu. Thanks Lauren. There are some really great
oering wishlists. It is an eective way to reduce shopping cart channels in that list.

A Year of #PartnerPicks: Inspiration for Your Next
Ecommerce Project
1 comment 23 days ago
Marcus Johnson-Smith Thanks for the insight. I'm a shopify
user as well. Building a brand is extremely important in growing
Introducing BuyButton.js Shopify's New JavaScript
Library
18 comments 2 months ago
Tony Lewis How about the only thing that will forever prevent
me and many other people from EVER using this.. Ability to

Subscribe d Add Disqus to your site Add Disqus Add Privacy

Grow your business with the Shopify Partner Program

Learn more

About Careers Press and Media Enterprise Sitemap

ONLINE STORE POINT OF SALESUPPORT SHOPIFY

Sell online Themes Point of sale 24/7 Support Contact

Features Shopping cart Features Shopify Help CenterPartner program

Examples Ecommerce hosting Hardware Forums Afliate program

Website editor Mobile commerce POS software API Documentation App developers

Online retail Ecommerce software Free tools Investors

Ecommerce websiteOnline store builder Blog topics

Terms of Service Privacy Policy