• by Melanie Courtright , May 3, 2011

At a conference I recently attended, each session began with a two-minute "rant" by someone who was willing to say, "I've had enough of this and I'm not going to keep quiet anymore." If I were to rant, it would be about online survey cheaters. Since I don't rant well, my energy is better used sharing what we can do to keep these people out.

I like to call this "Quality BlackJack" because there are 21 areas where you should implement quality controls. If you do so, and improve those controls continually, you can win the Quality Jackpot. You may not get to Fiji with your winnings, but you'll win peace of mind.

Sustainable data quality isn't about a cool new technique (though some are used) or the latest technology (though that can help). It's an end-to end monitoring and vigilance process, with metrics all along the 5 Rs of sample: Recruitment, Registration, Respondent Management, Research, and Rewards. It requires checkpoints and measurement at every point in the survey lifecycle, along the way. Here are the things you, or your partner, should be monitoring -- day and night.

Recruitment

1. Recruitment Source vetting: Initial testing of demos, attitudes, and behaviors of every source used to drive traffic, and regular interval measurement to ensure consistency.

2. Traffic metrics: Volume of traffic coming from every source, by demos, ensuring predictable volumes. Watch for shifts in data quality, minority representation, or technology ownership, to name a few.

3. Publisher Comparisons: Web sites and memberships should and will have unique characteristics. Look at the unique attributes of a traffic source, and figure out what that means to your sample frame and the resulting data.

4. Blending Strategy: Using all of the information above, decide: what's your blending strategy, and how do you ensure consistency over time?

5. Diversity & Breadth: To offset bias and increase representativeness, it's crucial to have a really broad set of sources that drive people from all walks of life. Think beyond just demographics to psychographics. You won't find everyone you need on a single of a few sites. They're in remote corners of the web, and you have to reach them where they live.

Registration

6. Captcha: Tools that require human eyes and fingers, along with interpretation and logic, to register.

7. Digital Fingerprinting /Geo IP/Proxy detection: Tools that look at computer identities, and the network path they came from, that reach beyond deletable cookies and survey tags.

8. Third-Party ID Solution: External identity validation, ensuring people are honest about their identity and age.

9. Email/Password Scans: Accounts with the same or similar email addresses or passwords are a red flag for fraudulent accounts.

Respondent Management

10. Profile Traps and Consistency Checks: Do people overstate illnesses, list too many ethnicities in an attempt to qualify, have data that is inconsistent with previous questions or visits? Are they paying attention and being truthful?

11. LOI Scans: Watch speed in your own surveys, and have clients send speeding information back to you so repeat offenders can be flagged.

12. Client Survey Invalids Rules & Scans: For all clients who use data cleansing and traps, request as much information back as possible, and in real time where feasible. Any time you see a daily increase in client invalids, investigate immediately.

13. Automated Red Herring and Trap Question Battery: By now, every sample company should have trap questions built into the system, randomization and intelligence. It shouldn't be manual, and it shouldn't be predictable, or it won't work.

14. Sampling Protocols/Rules: One of the most important -- and over-looked -- steps in the process is rules and standards around the actual sampling. How are the invitations pulled? Is there consistency between PMs? Between waves?

15. Bot Prevention: This takes skill, stealth, and persistence. You need tricks up your sleeve. Call me sometime if you'd like to know more.

Research Management

16. Design Partnership with the Client: We are in this together, so let's work as a team to reduce length, increase engagement, and make the flow work. An important part of data quality is keeping the members that provide meaningful data.

17. Member Services Approach to Problems and Complaints: Track every interaction with members, and handle their complaints quickly. Look for problem themes and use them to improve the systems and the surveys. Watch for frequent complainers, and use it as a red flag.

18. Replicable Survey Assignment Process: When using a router, be sure that the routing system doesn't introduce bias. Routing should be random and replicable.

Rewards/Redemption

19. Address Collection Verification: Collect addresses, standardize them, and validate them.

20. Community Aspect and Sharing Survey Results: When possible, share survey results with members as part of their reward. Reinforce the importance of their participation and their response quality. Help them become passionate about their involvement, and make them feel part of a community of people.

21. Reporting of redemption anomalies: Watch reward redemptions for unexpected changes. Shifts in incentive choice or a sudden increase in redemption can be important indicators of a potential threat.

 

If you watch all 21 steps I've outlined for variances and anomalies, you will notice shifts when they happen, rather than after they've impacted the data, and you'll be able to intervene and make changes. That's where technology steps in, to implement gates and solutions. Watch and react, every day. You have to stay watchful at every point in the survey and respondent process. You can't rest. You can't get comfortable. What worked yesterday won't work tomorrow.

So the next time someone discusses their data quality initiative and elaborates on the one or two things they have in place, I hope you'll put out an invitation to play "Quality Blackjack." Pull out this list and walk through it, asking for details on the approach being taken. It all matters, and it makes a difference in hitting the jackpot.