Data tables can be intimidating. A questionnaire of moderate length can have hundreds of columns and thousands of rows to wade through. At the same time, tables generally incorporate extremely valuable features that save researchers a lot of time. But only if you use them wisely. Here are some tips to help you use data tabulations more effectively.
Data tables apply significant testing to every possible combination of variables not because those comparisons make sense or are important, but rather because the software is following the algorithm it was programmed with. Researchers must determine in which cases each test actually makes sense. Does it make sense to run a chi-square with a continuous age variable? Generally, no. Does it make sense to calculate a correlation with gender? Generally, no. Though the algorithms will generally use the most appropriate test, it is the responsibility of the researcher to ensure the algorithms properly apply the rules. This means quantitative researchers must understand which tests are appropriate in which situations.
It is technically possible to run significance tests on a sample of two cases. Statistical algorithms are designed to account for sample size. However, while the software might warn you that the sample size is small, it’s the responsibility of the researcher to determine what constitutes a “too small” sample in each situation. It’s very possible for 30 to be too small in some situations and 100 to be too small in others.
Traditionally, a p-value of 5% is used to decide whether a statistical result is significant. Of course, you might decide to use 1% or 10% for your research depending on how much risk you want to take. It does mean that across all of the tests run on your set of tables, a percent of the statistically significant findings will be significant purely because of chance. Not because you discovered the most important finding of the decade, but simply because of chance. Which ones? The only way to truly know is to replicate the results several times and identify those that are consistently significant. In other words, don’t assume any of your results or conclusions are absolutely, perfectly correct. Ever.
Statistical significance will never tell you if a result is meaningful. In fact, some researchers believe that if a statistical test is necessary to determine whether something is worth paying attention to, then it’s not worth paying attention to. We’ve already seen that some tests will be significant due to chance. For example, Spurious Correlations keeps a running tab of correlations that are significant purely due to chance. It’s always up to the researcher to determine whether the relationship is meaningful and worth pursuing before considering what a statistical test says about the odds.
Running thousands of statistical tests should never be a substitute for thinking. Good researchers take the necessary time to think carefully about why they’re doing the research and what they want to focus on. They create hypotheses and carefully design research plans to confirm or deny those hypotheses. These processes help to eliminate unnecessary research and save time and money.
If you’d like help understanding your tabulations, please get in touch with us!
You might like to read these:
