A/B Testing 101: A Crash Course on Optimization

Without A/B testing -without experimentation- you have no concrete data to drive real growth.

What is A/B Testing and how does it work?

A/B tests are experiments in which two versions of something are compared against each other to see which performs better.

You can A/B test almost anything. Whether it’s an ad, a banner, a button, a landing page, a website, or an app –  it can be A/B tested.

When audiences visit your site or app, they are randomly split so that a percentage of users are shown version A, while others are shown version B. Statistical analysis will reveal which version gets the best results.

Image source: Visual Website Optimizer
Image source: Visual Website Optimizer

The A/B method lets you test very specific changes to your website, then collect and analyse the data so that the impact of the changes can be measured and lead to data-driven optimization.

The changes can be as simple as a headline (colour, wording) or as extensive as a complete redesign of the page. Then, half of your traffic is shown the original version, while the second half is shown the modified version to see which one gets the results you’re looking for – revenue, signups, downloads – whatever you’re trying to do with your website.

Why Should You A/B Test?

A/B testing may sound simple but the insight gained can lead to significant measurable improvement of the user experience and performance. When it comes to making guesses or assumptions about human behaviour – specifically, how people will interact with your site or app, the results are often unexpected and can only be uncovered through actually testing.

After testing one variable at a time – the cumulative effect of winning changes leads to tremendous boost towards your desired outcome or goals – such as conversion rates.

In addition, the overall spend on marketing can actually be decreased as the elements of the user interface become as efficient and effective as possible – especially when it comes to getting new customers. A/B testing also proves useful to help product developers and designers to optimize other goals such as user engagement, and in-product user experience – to name a few.

Here’s another example of why you should A/B test (for more examples, go to www.behave.org)  – the Citizens Bank experiment:

Should your call to action stress urgency or important information? 

Key Performance Indicator (KPI): Unique Open Rate (unique opens/net delivered)

Traffic Source: Existing banking customers. Traffic to site split 50/50 between version A and version B.

Goal: to determine which emotional language subject line will convert better – informative or urgent.

Difference between versions:

Version A: Urgent email subject line: “Act now – get help with selling or buying a home” 
Version B: Informational email subject line : “Important information about selling or buying a home”



Which version do you think won?

The Winner: VERSION B!

Compared to version A – the informational copy increased open rates 49.2%, at 99% confidence.

Why? Perhaps because urgency in subject lines is a widely used marketing tactic – one that has lost its effectiveness due to overuse. This particular tactic is an old trick – marketers are simply trying to trigger our impulsive instincts and motivate us to purchase or engage in some other call to action.

(Source for Citizens Bank example: BEHAVEhttps://www.behave.org/case-study/call-action-stress-urgency-important-information/)

Here’s another example from Highrise, a CRM software company:

Highrise’s Headline & Subheadline Test:

high-rise-a-b-testHighrise tested different headline and sub headline combinations to see how it affected their sign-ups. The test showed that the variation telling visitors that the sign-up is quick produced a 30% increase in clicks. It was also the only variation with an exclamation mark.

These examples show how A/B testing little changes can lead to big wins. So, what’s the process?

The A/B Testing Process

  1. Collect data: First, you must analyze and gain insight into where the bottlenecks are – such as pages with low conversion rates or high drop off rates and other user metrics such as page views, time on site, impressions, clicks, CPCs, and many more. Here are four tools to help gain this insight:

This information will help to set specific and measurable goals.

  1. Identify Goals: Figure out what metrics you will use to determine whether (or not) the changes get better results than the original version. Goals can be anything from website visit duration, email campaign signups, viewability of ads – the possibilities are vast and it is important to explore several possibilities before getting too focused as you risk missing the best possible solution. A helpful tip to start off: visit your website and jot down all the actions visitors can perform to create a list of possibilities.

While narrowing down to goals remember that less is more – keep focused and reduce choices so that you’ll get the most accurate results from testing. A straightforward goal example could be to simply optimizing engagement through a 20% increase in email list signups.

  1. Generate Hypothesis: Now that you know exactly what you want to improve, you can begin hypothesizing about what changes you think would make the biggest impact and then test. If your goal was to get more people to sign up for an email list or form – you could try something as simple as removing a few fields to make the process as painless as possible for the user to encourage more signups. We all know that attention span is short and that no one likes filling out lengthy forms (at least initially while to payoff is unclear). Or, what if your original call to action wasn’t as motivational as expected and your goal is to increase the response rate? For example, you want people to donate, so you might hypothesis that changing the wording from your current call-to-action “contribute” to “please donate” would be more engaging and direct.
  1. Generate Variations: Here’s where the A/B testing software comes in, which allows you to actually make the desired changes to your website or app and have the ability to see exactly what it will look like (especially helpful when changing colours or moving elements around) . A/B testing software resources include:
  • Optimizely optimizely.com (free trail for 30 days, affordable and easiest to get started, real-time support, multi-functional)
  • Visual Website Optimizer vwo.com (free trail for 30 days, feature rich, includes idea generation tools)
  • Unbounce unbounce.com (focuses solely on landing pages, offers 80+ pre-designed landing page templates)
  • KISSmetrics kissmetrics.com (premium, starting at $150 per month with full year commitment, offers advanced abilities, loads of data)
  1. Conduct Experiments: Let the testing begin! This is the part where visitors will be randomly sent to either the original version or the modified version to test which performs better. The way users interact with each version is monitored and compared to determine how the changes impacted the experience.
  1. Analyze Results and draw conclusions: You’re A/B testing tool will organize the data for you so you can see how your test version did compared to the original. Hopefully your test result support your hypothesis, but if not then there is still more hypothesising and experimenting fun to be had!


Every business, website, campaign, app, is different and A/B testing shows that common marketing wisdom often doesn’t apply. The advantages of improving conversion rates by A/B testing are well worth the relatively low investment of time and money.

See below for more surprising A/B testing discoveries that will make you want to test everything!




Want to learn more? Connect with our team at sales@clearpier.com

3 Key Takeaways from Digiday Publishing Summit 2016 in Key Biscayne

Digiday Publishing Summit Addresses the Current State of Digital Publishing

Earlier last week, over 400 leading publishing executives and tech providers gathered in Key Biscayne, Florida, for the Digiday Publishing Summit. The conference featured 38 speaking sessions by a diversity of industry leaders, with over 170 publisher VIPs in attendance.

Our team conferencing it up at #DigidayDPS #ClearPier @jaypkim6 @gage_mair @ben.mair @digiday @digidayevents

A photo posted by ClearPier (@clearpier) on

Throughout the three days of networking, sessions, meetings, and social events there was a genuine community feel unified by everyone’s sincere drive to improve the digital publishing experience. Conversations centered on four main hurdles: solving revenue challenges, finding effective distribution strategies, keeping up with technology, and getting the content right to fit each distribution channel.

1. Publishers can best monetize their revenue generation by combining strategies

Meeting revenue goals has never been so challenging in an industry that takes place in real time. There is no clear formula, and the factors are so dynamic that it’s a risky business often requiring trial and error, and a whole lot of adaptability. Speakers at the conference talked about the success of their revenue strategies:

Paid Subscription Models

  • Paul Rossi, President, The Economist, led a speaker session at the DPS about how the economist has long embraced the paid subscription model
  • Their success is a result of targeting niche audiences

Branded Sponsored Content

  • Joe Speiser, co-founder & CEO, Little Things, advocated for branded content
  • Little Things is a lifestyle publisher that engages 50 million visitors each month
  • They specialize in both original and branded content and employs social, especially Facebook to get reach
  • The result? High engagement and 75% traffic coming through Facebook with no subscription

Ad Monetization

  • In a panel at DPS, a partnership approach was highlighted as being critical bringing the buy and sell side together
  • Advertisers/agencies and publishers are coming together to maximise opportunities, often through custom executions
  • To facilitate these deals and boost revenue, publishers without the technical know-how are opting to work with technology providers and PMPs that will work with them exclusively to get the demand they need

2. Stay adaptable and keep your approach diversified to reach key audiences.

At the conference, conversations about reaching audiences focused on knowing how the audience consumes content. This information guides the development of effective distribution strategies, increasing reach, and finding quality opportunities to target audiences.

  • John Harris, publisher & editor-and-chief at Politico led a talk about “Building a balanced audience portfolio”
  • His message: relying too heavily on any one platform is dangerous because if the trend fades then the resources invested are lost
  • Diversification in terms of reach and building content to be experienced across platforms is another major key tip
  • For example, the use of video is rapidly growing and there’s pressure to understand and deliver effectively

3. Maintain your audience base by building trust through quality content

Quality content was reiterated by numerous speakers at Digital Publishing Summit.

“Timely, relatable and relevant,” these are the keys for engagement in social according to Rajiv Mody, VP of Social Media at National Geographic. Moday’s claim is backed by the fact that National Geographic consistently ranks No. 1 for total fans and engagement in the magazine industry across major social media platforms. It is also the most followed non-celebrity brand in the world on Instagram.

But in addition to quality content, Keith Hernandez, president of Slate, also advised audiences to explore new mediums. Slate specifically invests in podcasts rather than video and it has paid off – podcast audiences for the popular publisher tripled in 2015. Podcasts allow a different type of engagement and user experience – more of an in depth conversational delivery can be achieved through audio and has proven to provide the ideal user experience for a current events, politics, and culture content that Slate is known for.

But when it comes to video – a rising format in 2016 – Laura Buchman, VP of Publisher Platforms, tells us video is powerful across screens. She believes it’s crucial to develop, distribute, and monetize content through an all-screen lens. Buchman challenges publishers to consistently produce high quality video especially as audiences grow and expect fresh content on a daily basis. But there are set backs: the cost of video production are huge as is the pressure to build long term consistency rather than viral hits.

As we look forward to 2017, video has been reiterated as the format that poses some of the biggest challenges for publishers. Until the next Digiday Publishing Summit, we’ll have to watch the landscape and see how publishers and advertisers alike are keeping pace.

For more live-posts from the conference, check out our Twitter, Facebook and Instagram.

Want to learn more? Connect with our team at sales@clearpier.com