# Trust and models By:: [[Brian Heath]] 2022-09-02 Here is the situation: An analyst walks into a room with some business executives, presents a model that shows how a metric of interest will likely change in the future, and then someone asks, "How good is this model?" In other words, they are asking if it can be trusted. There are two dynamics at play here. First, there is the validity of the model, which is likely the first thing you thought of if you are analytically inclined. So, is the model valid at representing the thing of interest from a mathematical standpoint? The second dynamic is whether the analyst who built the model can be trusted. Which of these two dynamics is more important? Without a doubt, it's whether the analyst can be trusted. To show that the trustworthiness of the analyst is the most important part of "how good is this model," let's start with the black magic and hand waving that is mathematical model validation. There is [a lot to say here](https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=2121&context=etd_all), but for the sake of [[simplicity]]: there is no such thing as a model that fully represents the system of interest. Models cannot represent entire systems accurately and there is no way to prove it even if you think otherwise. The use of statistical validation [[metrics]] like p-values and other validation approaches are only attempting to sell a person that a model might be good, not whether the model is actually good. [[George Box]] is often quoted right about now when he said "All models are wrong but some are useful." Most people focus on the models being wrong, but I think the usefulness of models is the more interesting bit. A model is useful if it helps someone make a better decision. In our head we have a very simplified model of the sun rising and setting, so that helps us decide whether it's a good idea to grab a flashlight. This model is wrong in a lot of ways. Yet, it's a useful model for this decision and we've seen it work many times. We trust that this wrong representation of the world is good enough for this decision and this is at the heart any time someone questions you about how good your model is. Everyone intuitively knows that your model is wrong, but its usefulness is all about whether you are a reliable source. Utility eats accuracy for breakfast, lunch, and dinner, so build trust before you build your model. #### Related Items [[All Models are Wrong]] [[Models]] [[Trust]] [[Accuracy]] [[Validation]] [[Statistics]] [[Decision-making]]