Statistical Significance

What is Statistical Significance?

Statistical significance is the likelihood that a relationship between two or more variables is caused by something more than just random chance. In other words, it is the likelihood that the difference in conversion rates between any variation and the baseline is not a random occurrence.

When testing a hypothesis, the result of an experiment is thought to have statistical significance or to be statistically significant, meaning results are likely not caused by chance for any given level of statistical significance. When you’ve found the statistical significance level, it reflects the risk tolerance and confidence level of your hypothesis.

For example, say you run a split-testing experiment and find a significance level of 95%. This means that if you determine a winner within the split test, you can be 95% confident that the results you have observed are actually real and not an error caused by random chance.

A 95% confidence level means there is a 5% chance you could be wrong about your results. Thus, statistical significance allows you space to make better-informed decisions about marketing practices for your business.

What Does Statistical Significance Mean in Digital Marketing?

For digital marketers, statistical significance is important and necessary because it is a way to mathematically prove that a certain statistic is reliable.

When you’re making decisions based on the results of the experiments you’re running, you want to be sure that a relationship and certain level of certainty exists. Without it, the moves you’re making can be harmful to your business and you may not even know. While 100% certainty is not attainable, statistical significance gives you a certain level of accuracy and certainty in your marketing decisions.

For this reason, online web owners, advertisers, and marketers have all become more and more interested and invested in making sure that their split testing experiments (whether they’re for conversion rate a/b testing, email subject line tweaks and changes, or blog title changes) get statistical significance before they jump to conclusions and make any big decisions.

Why is Statistical Significance Important?

In business, making a wrong move is not usually a minor mistake. Missteps can be detrimental if you aren’t careful and they can cost you a lot of time and resources to fix. With this in mind, statistical significance is important because it provides marketers and business owners with confidence that the changes they are making to their website or app will actually have a positive impact on their conversion rate and other pertinent metrics.

Metrics, numbers, and statistics can fluctuate drastically from one day to the next, but statistical significance offers a strong mathematical foundation for making sound business decisions and avoiding false positives in one fell swoop.

Key Variables for Statistical Significance

When looking for a statistically significant result, you must be aware that any result you find is not attributed to chance. Rather, all statistically significant results depend on two key variables:

Sample Size

This refers to how large or small the sample for your experiment is. When your sample is large, the more confident you can be in the result of the experiment. For example, if you are running tests on your website, the more traffic your site receives the sooner you will have a large enough data sample to return results that are statistically significant. If your sample sizes are too low, you’ll run into sampling errors and find that your results aren’t as solid or trustworthy as you need them to be. Larger sample sizes lead to more accurate results – keep this in mind before you begin any type of testing.

Effect Size

The effect size refers to the size of the difference in results between two sample sets. In short, effect size indicates practical significance. For example, if you are working with small effect size, you will need a large sample size to help determine whether the difference is significant or just due to chance. On the other hand, if you notice a large effect on your numbers, you can then confirm with a smaller sample size with a higher degree of confidence.

In addition to the effect size and sample size, you have to think about the importance of randomized sampling. If the traffic to a website is split perfectly between two pages but the sampling itself isn’t random, it can actually lead to errors due to differences in behavior in the sampled population. Let’s spell it out like this:

Say you have 200 people visit a website. Each male visitor is shown one version of a webpage and each female visitor is shown a different version. In this scenario, a comparison between the two groups is not possible because the traffic is split 50/50 down the middle. Ideally, the difference in demographics should introduce variations in the data.

For this reason, a truly random sample is needed to determine that the result of the experiment has statistical significance. With this in mind, you can still have 200 people visit a website, but if the groups are truly random, they will not split evenly or easily, offering natural variations in the data which leads to a statistically significant result.

Statistical significance is an effective way to bring some confidence into the data you work with every day as a business owner. The more sound your decisions, the better your business is over time. To help keep your business in a vein of good decisions and proper care, consider working with Directive to ensure that your current website, blogs, and resources are on par with your business goals. Get a free proposal with us today to learn how you can improve your site even more.