Our UK Head of Insight Darren Lewis wants to start an industry wide conversation
It’s OK, I’m not going to revisit GDPR. We need to talk about data provision rather than data protection. Specifically, who is providing the data that is the lifeblood of our industry?
At Kadence we’ve always been proud of our data quality. We have our own online survey software and an in-house data analytics team who are obsessed with quality, so we can conduct enhanced quality control checks that may not be available to everyone.
We have our own B2B online panels and we also have access to high quality consumer panels in Asia through our company network. For consumer surveys in Europe and North America, however, we buy our share of online panel samples and over the years we thought we’d seen it all.
‘Speeders’ and ‘straightliners’ were routinely caught and immediately removed from our data, as were poor responses to open ended questions. There were even trends to these poor responses. Some US panellists were so anti-Obama they couldn’t resist telling us, in no uncertain terms, during FMCG concept tests, while more recently a number of panel respondents became fluent in Latin – a dying language for us humans but thriving among the bots. Always removed and reported back to the panel providers, no exceptions.
More recently we noticed some panel respondents becoming better at playing the game. Mainly straightlining through attitude statements and with open ended responses that could be genuine, i.e. ‘I like’, although this response was usually repeated throughout the survey. Removed. And reported. Again.
But then a few weeks ago we had an urgent request for 250 interviews among a sample of UK consumers, a typical online panel survey. The data generated by the sample was genuinely shocking.
Our IP address checks showed over 50 interviews had been completed by just 13 devices. One device had been used to complete the survey 11 times. Our ‘UK consumers’ were completing the study from far flung locations, particularly India, Netherlands and Pakistan. Again these were identified through IP address checks. A number of respondents completed the study after they had earlier screened out due to ineligibility. They just kept coming back to the survey until they found a way through.
Eventually we took a decision to scrap all the data for that survey.
It felt like a watershed moment for us. We were able to find these issues and remove the data because we conduct these extra checks, but it was extremely worrying for us all. How common are these practices? How big an issue is it for the industry?
There has always been a trade off with online panel samples; clients need data quicker than ever before and budgets are tight. We have become a data heavy industry, quantity and speed can seem to be more valued than quality. But are we willing to trade off quality? The answer was obvious, for us at least. No.
So, how do we solve this issue?
In the short term we sought reassurance from our main panel providers about the quality of the sample we receive, to be certain that respondents were real people, with real opinions that they wanted to share.
With traditional methodologies of Face-to-Face and Telephone this wasn’t an issue. We spoke to real people, often interviewing them in their home or speaking to them on a landline. There was proof that the person existed and we had a way of uniquely identifying them. Now, we can’t just go back to the old ways and abandon online surveys, but we can take some traditional elements and replicate them.
So, after discussions with our panel providers we instigated a new policy. We will now only use sample from panellists whose identity can be verified using offline means. Effectively, this is proof of name and address from the panel source, or proof of a bank account that is only attributable to one panellist. No proof, no sample.
What does this mean for us? It means we have significantly fewer panellists to choose from, online surveys may take longer to complete as a result, we may even see our costs rise in the long run. But these are necessary actions to ensure confidence.
In the longer term, we all need to have an open and honest conversation within the industry. Agencies, panel companies and clients. We are all responsible for the current situation, we all need to work together to resolve it.
- Panel companies must take their share of responsibility for providing samples with dubious panellists and not doing enough to check their validity or remove them.
- Agencies have gratefully accepted the lower prices for panel sample and not done enough to insist on higher quality. Some may not even be aware of the issues.
- Clients must understand that the cumulative effect of the drive for lower costs and faster results has exacerbated these issues.
We know there is pressure to deliver on all of us, but we must be sure that our data is 100% genuine if we are to have a future as an industry. Faster and cheaper? Yes, but it also has to be better.
So, we will not compromise on data quality and we’re going to have more of those open discussions with clients, suppliers and our colleagues within the industry. It’s too important an issue not to talk about it.
Now I think about it, maybe this post is about data protection after all.