#42 Perfect Data is a Myth

Data quality is important because we need accurate and timely information to manage services and accountability, good information to manage service effectiveness, to prioritise and ensure the best use of resources. Yet some say that nothing less than perfect data is enough. That requirement for data must be busted. There isn't any perfect data in real life cases and business.


Data will always have flaws, gaps and even mistakes. That does not mean we should not put efforts in enhancing data quality. definition of data quality is data that’s fit for purpose.


The reason data quality gets so much attention is when bad data gets in the way of getting the job done. If I want to send an e-mail to 10,000 customers and one customer’s zip code is unknown, it doesn’t prevent me from contacting the other 9999 customers. That can amount to what in any CMO’s estimation is a very successful marketing campaign. The question should be: What data helps us get the job done?


Data that’s 80%+ accurate may be enough for many operational or tactical decisions. They may only need a hint at the direction that data is taking overall for it to be valuable, and they may only need it for this instance. Immediacy often trumps purity.


Aiming for 100% perfect data is also economically stupid. The closer you get to 100% the harder and more expensive it becomes (Zeno’s paradox). Then you need to maintain the 100% perfect data as it regenerates and amount grows. Sounds like a fools job to me. In this case the aphorism that “perfection is the enemy of progress” is really true.


Next time someone yells, our data is not perfect, give them ideas from above to think through.