# Data and Instinct By:: [[Brian Heath]] 2022-09-09 Within the analytics field, trusting your gut is often viewed negatively. This sentiment is best summed up by statistics pioneer [[W. Edwards Deming]]'s famous quote: "In God we trust, all others bring data." There are many ways one can interpret what Deming meant by this. If you interpret data as any [[information]] or observation, then this quote amounts to making decisions based on observations. This broad definition makes Deming's quote essentially a tautology. There is no other way for humans to make decisions. Everything we learn throughout [[life]] is via observation of "data," thus our instincts are data-driven decisions by biological sense and response. This includes the instincts we are born with because at some point a data observation was made and became encoded into our DNA. However, given his statistical and business background, Deming was most likely highlighting the importance of using statistical data in business decision-making. Thus, the battle against instinct began in the analytics community. Certainly, more information can result in better decisions, but there are at least two scenarios when gut instinct is valuable. The first is when statistical data isn't available. Analytical techniques exist to solve this problem by generating the data, but using these techniques requires just as much art as science. Whenever you venture into the world of art to generate imaginary data, you are probably violating Deming's edict and using some gut instinct. Only through hand waving and black magic is a skillful analyst able to reconcile and resurface the data-is-greater-than-instinct principle. The second scenario is when a data-driven decision isn't worth the time and effort. It could be that the decision is extremely low risk, such as whether to open a window. In this case, experimentation is often the easiest path to a good outcome. Or, it could be that cost of fully generating a statistically valid decision isn't objectively worth it. For example, deciding who is the most efficient driver to reduce the cost of gas. You could run many experiments, but you'll spend a lot of additional money on gas testing and still not get anywhere. Here gut instinct and group decision-making must kick in. So, gut instinct has its place alongside data-driven approaches. However, given that the dynamic between these two methods is more fluid and complex than the field would like to think, what happens to an analyst who becomes an expert in their field? An expert has accumulated so much data within their lifetime that certainly the break-even point between the gut and data-driven methods has changed. For the expert, it will eventually become more cost and time effective to use gut instinct and abandon data. This creates an obvious paradox for an analytics expert. You may still use data approaches when appropriate, but instinctual decisions will begin to dominate. As a result, you'll either be viewed as a wise sage who is a bit weird or you'll be the outcast who lost their way and betrayed the field's ideology. However, this could simply a symptom of getting older. Causation and correlation are hard to figure out. #### Related Items [[Analytics]] [[Data-Driven Decisions]] [[Instinct]] [[Expertise]] [[Statistics]] [[Ideology]]