There are two main types of samples and researchers are regularly at odds with each other over which one is better. In reality, neither is better. Each one is better suited for some needs and less so for other needs. Let’s get a better understanding of what each of these sample types are and when you might have a preference for one or the other.
What is a probability sample? In the most simplest terms, probability samples are used where the probability of each unit being selected is known. For example, if the entire population is 100 people, and you put each of their names in a hat and select ten names randomly, you know that each person’s chances of being selected are one in ten. This is a probability sample.
- What is a real-life example of a probability sample? Imagine an office at packaged goods company. You know exactly who belongs that population and exactly how to identify them. You simply need to go to the Human Resources department and ask for a list of every employee – the population. You can then use names, numbers, email addresses, or something similar to randomly select a smaller number of people to form a sample. If there are 1000 people in the company and you need a sample of 200 people, you know that each person in the population has a 2 in 10, or 1 in 5, chance of being selected.
What is a nonprobability sample? On the other hand, a nonprobability sample is where you don’t know the probability of each unit being selected. For instance, maybe the entire population is 100 people but you could only find 95 of the names to put into the hat. Or, you found 100 names to put in the hat but you don’t know if the population is 100 or 110 people.
- What is a real-life example of a nonprobability sample? Imagine you manage a consumer packaged goods website where 10 000 people have signed up to receive coupons and samples from you. In fact, these 10 000 people do not represent all of your customers. Some of your customers never visit your website and some have visited but declined to sign up for your offers. If you wanted to research what your customers think about your newest product, selecting a sample of 1000 people from your email list to do so would reflect a nonprobability sample. Why? Because while you may have a list of some of your customers, you don’t have a list of the entire population of customers and so you can’t calculate each person’s chance of being included in the research.
Advantages of probability samples: Though probability samples are generally much more expensive and take longer to conduct, they allow researchers to generalize the results back to the larger population with a known chance of error. That’s why you’ll normally see the margin of error statement, “+ or – 3 points, 19 times out of 20.” Though often misused, that phrase is intended to only be used with probability samples.
Probability samples are most appropriately used in highly sensitive and serious situations, for example in life and death scenarios where the research will be used to determine survival rates, or to generate social and health policies that affect the health and well-being of people. When accuracy could mean that 2000 versus 2500 people will lose or gain a medical treatment, probability samples could be the more appropriate solution.
Advantages of nonprobablity samples: On the other hand, though nonprobability cannot generalize back to a population with a known chance of error, they are far easier to conduct, take less time to conduct, benefit from easier sampling, and are far more cost efficient. To attempt to compensate for sampling error, they are often designed to match a targeted population in terms of demographic details such as age, gender, and region. And, some of the risk can be preempted by using multiple samples, whether from different regions, different providers, or different times. Replication is always an admirable goal!
When extreme precision is not warranted, nonprobability samples have many good applications. Here are just a few:
Examples of nonprobability sample research:
- Targeting socially undesirable or illegal behaviours: In many countries, being gay is still a taboo. In others, being gay is outright illegal. As such, finding a comprehensive list from which to sample could be literally impossible. In such cases, probability sample will also be literally impossible and a nonprobability sample will be your best solution. It may not be a probability sample, but the information acquired will be far more valuable than having none at all.
- Pilot studies: The purpose of pilot studies is not to generate data that can generalize to a population, but rather to identify weaknesses within a specific test or methodology before launching to the full sample. For instance, a pilot test may reveal that there so many people are selecting the ‘don’t know’ or ‘none of the above’ options that you need to add more answer options. Or, that very few people recognize the brand names you’ve selected, or that no one is screening into the most important target group. Pilot tests alert you to potential problems before they become emergencies. Pretty good samples, rather than perfect samples, will help you get this work done quickly and at a lower cost.
- IHUTs, package, and sensory tests: The purpose of package tests is not to generalize to the population. Package tests help you understand a variety of things such as how difficult is it for people to open the package, particularly children, the elderly, or people who have disabilities. They help you understand whether the colours and design you’ve selected reflect the mood you intended to convey. They help you understand whether the package truly is unique amongst its competitors. Package tests need to alert researchers to the extremes, the worst case and best case scenarios, not provide generalizable precision.
- Eye tracking and heatmapping: The purpose of eye-tracking is not to generalize to the population, but rather understand the edge-cases of how people look at and pay attention to something. Knowing a general pattern of behaviour is the end goal, as opposed to knowing precisely how every single demographic and target groups consumes an image.
If you’d like help figuring out whether your research is best suited to a probability or nonprobability sample, please get in touch with us! We’d love to help you.
You might like to read these:
