Researchers might not seriously consider working with a particular panel because they believe it is too small. When some panels advertise their size in the millions, it can be hard to seriously consider a panel that advertises ‘only’ hundreds of thousands panelists. A decision to work with the larger panel seems obvious. Or is it?
To illustrate why panel size is often not relevant, let’s consider two hypothetical companies. MassivePanel says they have 1 million panelists. They launched a questionnaire to everyone on their panel and got 50 000 completes. This equates to a response rate of 5%. On the other hand, MaintainedPanel says they have 200 000 panelists. They launched the same questionnaire to everyone on their panel and also got 50 000 completes. This equates to a response rate of 25%.
Even though MassivePanel is 5 times as large as MaintainedPanel, it turns out that the effective sizes of the two panels are identical. There is no advantage to choosing the larger panel because the end result is the same. So what’s happening?
Panel rules are what’s happening. Most panels have rules about who they include when they define their panel size. MassivePanel might have decided that a panelist is ‘active’ if they clicked on a survey invitation in the past 12 months. A rule like that would result in a very large panel as many people would click on at least one email, even accidentally, over the course of a year. The survey completion rate would be low but the panel size would be large.
On the other hand, MaintainedPanel might have implemented a much stricter rule. They might have decided that an ‘active’ panelist is someone who has fully completed at least one survey in the last 3 months. A lot fewer people would meet this criterion but the panel would be full of people who are known to fully perform the research tasks asked of them. The survey completion rate of this panel would be high although the panel size would be small.
In other words, panel size is meaningless without an estimate of the research completion rate. The next time you’re considering which panel to work with, keep in mind that the starting panel size isn’t always the best indicator. The best indicator is whether you will receive the number of completed research studies you require.
You might like to read these:
- Your Plain Language Guide to Product and Package Testing
- 9 Essential Checks for a Questionnaire Pilot Test
With nearly 40 years of experience, Canadian Viewpoint is a field and data collection company that specializes in English and French offline and online services. We offer consumer and medical sample, programming and hosting, custom omnibus, mall intercepts, pre-recruits to central location, mystery shopping, site interviews, IHUTs, sensory, product, and package tests, discussion boards, CATI, facial coding, and other innovative technologies. Learn more about our services on our website. Canadian Viewpoint is a founding board member of CRIC (Canadian Research Insights Council) and named on both the 2019 GRIT Top 50 list of Emerging Players and the Women in Research shortlist for Best Places to Work.