10 Commandments of Analytics

Photo by Markus Winkler on Unsplash
  1. Correlation shows the way but doesn’t ‘causally’ lead it: Correlation doesn’t imply causation. Well designed controlled experiments are your friend to prove causation (with a quantifiable risk of being wrong)
  2. The signal always hides within and beyond the noise: Understand the patterns and variance of your data and metrics to be able to detect the signal and not fool yourself (and others) with the noise.
  3. You measure what you track and you impact what you measure: If you want to start measuring something, track it properly first and be aware of unnoticed and creeping biases (like when there is a bug with the tracking). A metric only captures what is meant to capture and misses the rest. If you can’t measure it, you can’t track your progress in it.
  4. Metrics have a lifecycle and sometimes an expiry date: Metrics need to evolve with the product, the business, the people, the knowledge and the tracking & analytical capabilities.
  5. Data-driven till the end: You don’t need to be 100% data driven the whole time. There is always a place for subjective interpretations and product decisions. Use this deliberately and harmoniously with data
  6. In data we trust. In how it is reported, a bit less: Data doesn’t lie but it can support your lies (deliberate or un-deliberate). This makes story-telling is a double edge sword. Use it responsibly
  7. Give me less data, please! Having more numbers/charts/stats is tempting. Resist it as long as it is not adding value. ‘Less is more’ applies here
  8. Statistical significance is binary, you either have it or you don’t: Don’t subjectively argument and discuss statistical significance. Discuss what happens before it (experiment design and power calculation) and after it (decision related to the experiment)
  9. Bayesian is our saviour: Bayesian is a double edge sword, using it calls for more refined intuitive understanding of Analytics. Only use it if you’ve mastered the previous points
  10. Your cognitive biases is the ultimate enemy: Even if you think that you’ve mastered the last points fully, stay humble and don’t trust yourself blindly. As humans stats/analytics do not come to us naturally, it’s a skill we build and keep on refining. Having a sparring partner to review your stats helps here.