The question is simple but the answer is complicated. I hope you’ll read through the entire post but you can jump to the bottom if you’re in a hurry for an answer. First, let’s understand the components that contribute to figuring out the best sample size. Second, let’s tackle a few examples. Here goes!
What factors contribute to choosing a sample size?
Some of these factors are interrelated but let’s consider them separately so that you can decide which ones are most relevant to your situation.
Your research objective. If you simply want to gain a better understanding of which product or feature or size people like more, you probably won’t need decimal place precision. Indeed, you might only need enough precision to say that 20% or 25%, not 21% or 24%, of people like a feature. On the other hand, if you need to predict future sales of your product to determine whether a launch is feasible, decimal places might be extremely important. Knowing that 20% vs 21.7% of people prefer a specific product could translate into thousands of dollars. Precision demands larger sample sizes.
The importance of the results. Some research projects are culturally or financially interesting but not necessarily life changing. Thus, it’s financially advantageous to know which scent of bar soap people wish to buy but it’s life-changing to know which advertisement will encourage people to get screened and treated for colon cancer. Research that will lead to life or death decisions needs larger sample sizes.
How much risk you’re willing to take. If your job and career depend on the accuracy of the research and you’re not willing to risk losing your job, then make sure your results are as precise and reliable as possible. Results from one small sample of a hundred people could easily change once implemented in the real world of thousands or millions of people. Risk takers can get away with smaller sample sizes.
How different do you think the groups will be. When you get to the stage of comparing two things, whether that means men and women, old and young, the blue box and the green box, the vanilla scent and the licorice scent, do you expect the results to be very close with 49% preferring one thing and 51% preferring the other? Or, do you expect there to be a clear winner with 70% preferring one thing and 30% preferring the other? Close races need larger sample sizes.
The number of groups you will compare. Do you plan to compare results between just two groups, for instance women and men, or do you plan to compare results among 6 or 10 or 12 groups? Thus, if you break the income variable into six groups, will you compare preferences among each of the income groups? Or, will you compare preferences among people broken out by age, gender, and income for a total of 12 groups? If one group needs 100 people, then 12 groups need 1200 people. Research with fewer groups can get away with smaller sample sizes.
Random chance doesn’t care about you. No matter how small or large a sample is, it will always be a small proportion of the entire population and you will never know for certain what the population answer would be. You can only strive to generate data that will allow you to make a more educated estimate. Random results can pop up anywhere, anytime, anyhow, and the resulting conclusions might surprise the heck out of you.
What is the margin of error?
Market researchers collect samples of opinions so that they can estimate the population of opinions. We know that results collected from samples aren’t perfect so we create a buffer around our results. This buffer is typically called the margin of error, and it can be determined statistically for probability samples. Here are a few examples.
- For a sample of 300 people, the margin of error around 30% is 5.3 points. In other words, we can be fairly certain that the population number is somewhere between 24.7% and 35.3%.
- For a sample of 300 people, the margin of error around 10% is 3.5 points. In other words, we can be fairly certain that the population number is somewhere between 6.5%% and 13.5%.
- For a sample of 800 people, the margin of error around 30% is 3.2 points. We can be fairly certain that the population number is somewhere between 26.8% and 33.2%.
- For a sample of 800 people, the margin of error around 10% is 2.1 points. We can be fairly certain that the population number is somewhere between 7.9% and 12.1%.
From these examples, you can see that larger samples allow you to be more confident about the precision of your results. And, you can see that the further your raw percentages are from 50%, the more confident you can be about the precision of your results.
So how many people should we test?
Alright, alright, alright. Let’s get to the answer you’ve been waiting for! Here are four simple scenarios comparing the results of one group to another. First, we estimate how different we think the results of the two groups will be. Alternatively, we estimate the magnitude of difference that we care about (e.g., we could decide that once a difference is more than ten points, we don’t need to get any more precise).
Then, we use statistical tables to figure out what sample size corresponds with the risk we’re willing to take.
In this case, someone who is uncertain about the potential results and not willing to take much risk would probably go with a sample size of 600 for each of the two groups for a total of 1200 people. On the other hand, someone who is willing to take some risk might choose a sample size of 100 for each of the two groups for a total of 200 people.
| 48% of people would buy Product A52% of people would buy Product B | N=300048% (46.2% to 49.8%) is statistically different from 52% (50.2% to 53.8%) |
| 45% of people would buy Product A 55% of people would buy Product B |
N=60045% (40.9% to 49.1%) is statistically different from 55% (50.9% to 59.1%). |
| 40% of people would buy Product A 60% of people would buy Product B |
N=10040% (30.1% to 49.1%) is statistically different from 60% (50.1% to 69.1%). |
| 35% of people would buy Product A 75% of people would buy Product B |
N=2535% (15.1% to 54.9%) is statistically different from 75% (55.1% to 94.9%). |
If you love this kind of math, have a look online for sample size calculators. There are lots of tools that will help you choose the precise sample size that will work for you.
We’d love to help you sort out sample sizes for your research. Please get in touch with us!
You might like to read these:
