How to Conclude CRO Tests

Lesson Preview

In this lesson, we’ll be walking you through how to conclude and keep track of your CRO A/B tests. 

Why is this important?

Learning how to properly conclude tests is critical in the progression of your overall CRO efforts. If you run an inefficient A/B test for 3 years with no clear winner, it would take you way too long to see any ROI from your CRO efforts. Also, if you don’t know how to measure the statistical significance of a test properly, you might end up choosing the wrong variant to move forward with. 

What is statistical significance?

Statistical significance is the probability that a relationship between 2 or more variables is caused by something other than chance. In other words, when you run a test that is statistically significant, you have a high enough confidence level that the results of your test did not happen by chance. If you have a test reach a confidence level of 75%, you have a 25% chance of being wrong in your assumption and a 75% chance of being right. That is why we recommend with our testing that you aim for a +90% confidence level so that your chances of being wrong with your assumption are slim. 

This may seem complicated but don’t worry! We’ll show you various tools in this lesson that will give you clear visibility into when you should end your test.

What you’ll need:

• Template: CRO Test Calculator & Tracker
• Free A/B Testing Calculator Tools:
  • http://www.abtestcalculator.com/
  • https://vwo.com/tools/ab-test-siginficance-calculator/
  • https://neilpatel.com/ab-testing-calculator/

The Process:

It is important to make sure that tests are concluded when they have achieved reliable results. If done too soon or without the right data, the wrong conclusion can be made which may cause further tests to continue in the wrong direction without the correct understanding of why this is occurring. When concluded correctly, further tests will continue in a streamlined process with an understanding of what effect each variable is having and why.

To do so, you can use this testing template where tests can be compiled, noted, and kept track of. You can also use this sheet to calculate your test’s p-value which is a metric to help you identify your test’s confidence level. Remember, you want to aim for +90%.

Below is a breakdown of how we would recommend you keep track and identify when to conclude tests.

Step 1: Starting to document your test

First, you’ll want to open up the testing tracker template and create a name for your test. Also input the start date of your test and the primary KPI (demos, purchases, downloads, etc.)

Step 2: Monitor Your Test Results

Depending on the volume of your website or campaign traffic, it may take a shorter/longer period of time to reach statistical significance for your test. With these factors in mind, here are some pointers:

• Set an approximate timeline for when you’ll need to check on your test results. 
  • Pick a timeline you feel comfortable with. If you are first starting out, you might want to check your test on a daily basis. If you know that your test will take a longer time, set up your own framework that allows you to check on your experiment consistently while maintaining efficiency.
  • We typically will allow a test to run for at least a week before digging into the data to see if we’ve reached statistical significance. Some tools like Google Optimize allow you to set automated notifications for when a test is close to reaching significance levels.
  • 
• Most CRO testing platforms, like Google Optimize, will have built-in Test calculators that will automatically run formulas and visually notify you in their dashboard if your test has reached statistical significance. If you are not leveraging a testing platform that has this, we recommend the following steps:

1. You can use the provided template to insert your traffic and conversion data to automatically measure your statistical significance level. Below is a visual example:

1. • Insert visits and conversions for both variants:

1. • The sheet is set up to automatically calculate your confidence level in Column P. If your test reaches +90% confidence, you can state your conclusion and conclude that the test is conclusive.

2.If you prefer to build your own test tracker, you can use the following free A/B test calculators to measure if your test has reached statistical significance: