Researchers can create the greatest discussion guide or survey. But it won’t matter if significant rigor is not put in place to assure that the right audience or sample is utilized. Use of convenience samples, poorly designed screeners and the often blind over-reliance on “cheaters and repeaters” can doom the best designed data collection instruments. One of the great benefits of having vertical category expertise, is our firm’s ability to easily identify the all too often scenario where a potential, but unqualified qualitative respondent has faked their way through an initial screener just to collect an incentive. Good researchers can first avert these scenarios by going back to questionable recruits and asking technical, subject specific questions in their re-screener. The rescreener can serve to validate the initial responses of those who look questionable. A cheater or repeater upon re-screening will often cancel last minute, give a wrong answer, or forget what they said in the first place. That saves both researcher and client, valuable time and money.
It’s both frustrating and amusing to observe vast differences between respondents who qualifies for a specific research study and know their subject matter compared to someone who is just trying to talk their way around a topic.
In our quantitative work, we also realize how important this is. That is why we deploy a variety of best of breed respondent quality checks, disqualifying respondents that demonstrate faults for straightlining or other patterned responses. We will also flag respondents whose completion time, deviates significantly from the expected and observed average survey length. Rotation and randomization of close-ended question choices eliminate ordinal biases. Finally, we can utilize “trap questions” strategically inserted within the survey to validate respondent engagement. In these ways, we are helping to fight against respondent fraud and gaining a true and complete understanding of our findings.