Good Data Better Than More Data

We live in an age that inundates us with more data than we have ever had before, says the report. but more data does not necessarily mean better decisions.

How six costly data mistakes can lead you astray, by William Gadea for Marketing Profs. 

In 2012, management consultancy CEB, now a subsidiary of Gartner, studied how more than 5,000 employees in 22 global companies used data in their decision-making.The researchers concluded that there were three categories of workers:

  • The visceral decision-makers, who followed their instincts rather than the numbers
  • The unquestioning empiricists, who followed the data slavishly
  • The informed skeptics, who considered the data, but used it with caution

The third category, those who used data with a measure of skepticism, balancing the figures with their own judgement, ultimately made the best decisions, but how can we take advantage of evidence without being led astray by numerical mirages? asks the report. Gathering data and learning from it can be such an effort that sometimes we cling too tightly, says the report, and too long to the truths we learn. Things change, and so what was true two or three years ago might not be true today.

What if in the last three years or so the returns of paid listings vs. organic listings on search has doubled or tripled? and organic click-through rate has fallen, asks the report. It's reasonable to look at your past-year performance to use as a baseline.

However, to do that you need to make sure that there's no variable that is skewing the result during the base period you're examining. 

Likewise, if you are looking at data retroactively and you are trying to see whether there is a correlation between an action you took and a result, check to see whether there are any lurking variables that might have also influenced the result.

Statistics allow you to come to a conclusion about a set of data from a random sample of its items. A mathematical formula that lets us calculate how likely it is that a sample will be an accurate reflection of the larger set; the output is called sampling error. There are various common misconceptions about sampling error:

  • Saying the sampling error is +/-3% does not mean that is as large as the error can be. Indeed, 5% of the time, the sampling error could be larger than 3%.
  • Saying that the sampling error is +/-3% does not mean that it is as likely to be 3% off than it is to be spot on. The probability curve peaks in the middle.
  • Saying that the sampling error is 3% does not mean that is the only way the data can be wrong. This article calls out some of the other ways!

Bad data might also be technical failure, human error, or bad protocols. When you look at numbers, think of how reliable the entire process of collecting them might be. And, sadly, numbers are often used to hoodwink us, says the report. If you're basing a decision on some data, and the source of that data is an interested party, you might want to take a closer look at, if not discount altogether, the representations being made.

Not measuring the right thing is the trickiest and most costly mistake to make.

Sometimes it's made because we want to fool ourselves or make ourselves feel better. Consider the so-called vanity metric. Is website traffic booming but conversions are flat? It's easy to emphasize the former rather than the latter. It might take a rationalization, but one can manage that, but sometimes we are not measuring the right thing because measuring the right thing is too hard... or maybe not even possible.

Customer satisfaction, for example, is really hard to measure accurately, even with established metrics, concludes the report. Data can be an enormous boon, but it’s important to know where it’s coming from  Measures like engagement are far easier to see and record, but which is going to be more important to the long-term health of the business, questions the report.

For additional information from Marketing Profs, please visit here.



2 comments about "Good Data Better Than More Data".
Check to receive email when comments are posted.
  1. Ed Papazian from Media Dynamics Inc, September 13, 2018 at 3:01 p.m.

    JacK, it is impossible to calculate the degree of accuracy---as most people understand the term----based on the sample size in a survey. All that these machinations tell you are the odds that you will get the same answer---a right or a hopelessly wrong answer---again if you conduct the identical study in exactly the same manner, with the same sample size, a second or third time. When it comes to real world accuracy, not consistency, the deciding factros are how good a sample you have in terms of representing the population at large and, especially, how the information is collected. For example the old Arbitron radio diary studies had the same degree of "statistical accuracy" based on their sample sizes as the more recent PPM measurements yet the latter showed 25-30% less listening per station---a huge difference--- and a substantial increase in reach per station. Clearly, the statistical accuracy calculations had no bearing in determining which findinds were" correct". According to them, both were "correct".The same thing applied to magazine audience measurement. The old Simmons "through-the-book" aided recall studies had the same sample size as the MRI "recent reading" studies---yet the latter found much, much higher audience levels. In both cases--radio and magazines---- the primary reason for the difference in results was how the "audience"  levels were determined, not the sample size.

  2. Marcelo Salup from Iffective LLC, September 14, 2018 at 8:25 a.m.

    Yes, "good" is good, definitely preferable over "more" but, "better" is better.

    Data that doesn't meaasure what you need to grow the business but what others have always measured is awful. It's just useless.

    I recently got a survey from Home Depot. Scale of 1 to 5. To the best of my ability, everything fell between 3.5 and 4.1. How is that helpful? It doesn't measure what I prefer in the store. It doesn't measure what drives me to go.

    So, yes, probably the survey will have an error margin of 0.00000000001% or whatever. And it will make some poobahs happy. But it won't give them intelligence. Just data.

Next story loading loading..