Key Variables for Statistical Significance
When looking for a statistically significant result, you must be aware that any result you find is not attributed to chance. Rather, all statistically significant results depend on two key variables:
Sample Size
This refers to how large or small the sample for your experiment is. When your sample is large, the more confident you can be in the result of the experiment. For example, if you are running tests on your website, the more traffic your site receives the sooner you will have a large enough data sample to return results that are statistically significant. If your sample sizes are too low, you’ll run into sampling errors and find that your results aren’t as solid or trustworthy as you need them to be. Larger sample sizes lead to more accurate results – keep this in mind before you begin any type of testing.
Effect Size
The effect size refers to the size of the difference in results between two sample sets. In short, effect size indicates practical significance. For example, if you are working with small effect size, you will need a large sample size to help determine whether the difference is significant or just due to chance. On the other hand, if you notice a large effect on your numbers, you can then confirm with a smaller sample size with a higher degree of confidence.
In addition to the effect size and sample size, you have to think about the importance of randomized sampling. If the traffic to a website is split perfectly between two pages but the sampling itself isn’t random, it can actually lead to errors due to differences in behavior in the sampled population. Let’s spell it out like this:
Say you have 200 people visit a website. Each male visitor is shown one version of a webpage and each female visitor is shown a different version. In this scenario, a comparison between the two groups is not possible because the traffic is split 50/50 down the middle. Ideally, the difference in demographics should introduce variations in the data.
For this reason, a truly random sample is needed to determine that the result of the experiment has statistical significance. With this in mind, you can still have 200 people visit a website, but if the groups are truly random, they will not split evenly or easily, offering natural variations in the data which leads to a statistically significant result.

Statistical significance is an effective way to bring some confidence into the data you work with every day as a business owner. The more sound your decisions, the better your business is over time. To help keep your business in a vein of good decisions and proper care, consider working with Directive to ensure that your current website, blogs, and resources are on par with your business goals. Get a free proposal with us today to learn how you can improve your site even more.