10 Commandments of Analytics
2 min readDec 21, 2021
--
- Correlation shows the way but doesn’t ‘causally’ lead it: Correlation doesn’t imply causation. Well designed controlled experiments are your friend to prove causation (with a quantifiable risk of being wrong)
- The signal always hides within and beyond the noise: Understand the patterns and variance of your data and metrics to be able to detect the signal and not fool yourself (and others) with the noise.
- You measure what you track and you impact what you measure: If you want to start measuring something, track it properly first and be aware of unnoticed and creeping biases (like when there is a bug with the tracking). A metric only captures what is meant to capture and misses the rest. If you can’t measure it, you can’t track your progress in it.
- Metrics have a lifecycle and sometimes an expiry date: Metrics need to evolve with the product, the business, the people, the knowledge and the tracking & analytical capabilities.
- Data-driven till the end: You don’t need to be 100% data driven the whole time. There is always a place for subjective interpretations and product decisions. Use this deliberately and harmoniously with data
- In data we trust. In how it is reported, a bit less: Data doesn’t lie but it can support your lies (deliberate or un-deliberate). This makes story-telling is a double edge sword. Use it responsibly
- Give me less data, please! Having more numbers/charts/stats is tempting. Resist it as long as it is not adding value. ‘Less is more’ applies here
- Statistical significance is binary, you either have it or you don’t: Don’t subjectively argument and discuss statistical significance. Discuss what happens before it (experiment design and power calculation) and after it (decision related to the experiment)
- Bayesian is our saviour: Bayesian is a double edge sword, using it calls for more refined intuitive understanding of Analytics. Only use it if you’ve mastered the previous points
- Your cognitive biases is the ultimate enemy: Even if you think that you’ve mastered the last points fully, stay humble and don’t trust yourself blindly. As humans stats/analytics do not come to us naturally, it’s a skill we build and keep on refining. Having a sparring partner to review your stats helps here.
