Data tables apply significant testing to every possible combination of variables not because those comparisons make sense or are important, but rather because the software is following the algorithm it was programmed with. Researchers must determine in which cases each test actually makes sense. Does it make sense to run a chi-square with a continuous age variable? Generally, no. Does it make sense to calculate a correlation with gender? Generally, no. Though the algorithms will generally use the most appropriate test, it is the responsibility of the researcher to ensure the algorithms properly apply the rules. This means quantitative researchers must understand which tests are appropriate in which situations.
It is technically possible to run significance tests on a sample of two cases. Statistical algorithms are designed to account for sample size. However, while the software might warn you that the sample size is small, it’s the responsibility of the researcher to determine what constitutes a “too small” sample in each situation. It’s very possible for 30 to be too small in some situations and 100 to be too small in others.
Traditionally, a p-value of 5% is used to decide whether a statistical result is significant. Of course, you might decide to use 1% or 10% for your research depending on how much risk you want to take. It does mean that across all of the tests run on your set of tables, a percent of the statistically significant findings will be significant purely because of chance. Not because you discovered the most important finding of the decade, but simply because of chance. Which ones? The only way to truly know is to replicate the results several times and identify those that are consistently significant. In other words, don’t assume any of your results or conclusions are absolutely, perfectly correct. Ever.
Statistical significance will never tell you if a result is meaningful. In fact, some researchers believe that if a statistical test is necessary to determine whether something is worth paying attention to, then it’s not worth paying attention to. We’ve already seen that some tests will be significant due to chance. For example, Spurious Correlations keeps a running tab of correlations that are significant purely due to chance. It’s always up to the researcher to determine whether the relationship is meaningful and worth pursuing before considering what a statistical test says about the odds.
Running thousands of statistical tests should never be a substitute for thinking. Good researchers take the necessary time to think carefully about why they’re doing the research and what they want to focus on. They create hypotheses and carefully design research plans to confirm or deny those hypotheses. These processes help to eliminate unnecessary research and save time and money.
If you’d like help understanding your tabulations, please get in touch with us!
You might like to read these:
- Why Is The Number Of Panelists Not The Best Criteria For Judging An Access Panel?
- What are the different types of marketing segmentation?
- Research and Analytics Dashboards: From Idea to Design
With nearly 40 years of experience, Canadian Viewpoint is a field and data collection company that specializes in English and French offline and online services. We offer consumer and medical sample, programming and hosting, custom omnibus, mall intercepts, pre-recruits to central location, mystery shopping, site interviews, IHUTs, sensory, product, and package tests, discussion boards, CATI, facial coding, and other innovative technologies. Learn more about our services on our website. Canadian Viewpoint is a founding board member of CRIC (Canadian Research Insights Council) and named on both the 2019 GRIT Top 50 list of Emerging Players and the Women in Research shortlist for Best Places to Work.
Follow us on Twitter or Linkedin, or sign up for our newsletter.