Why you can still use models, even if they don’t deliver certainty
You want to believe the science, but it’s hard. Earlier in the year, a COVID-19 model “predicted” over 2 million deaths in the U.S. This is far from our current reality. What a colossal error! Can we really trust this model? Can we trust any model at all?
In short, yes, we can still use models, but we need to be clear on their limitations. As Yogi Berra says, “It’s hard to make predictions, especially about the future.” Nobody knows what’s going to happen. Yet, we can still represent underlying phenomena in ways useful for decision-making. A good model will present a range of future scenarios and be clear about its limitations. It will help explore and prepare for the future with open eyes. It’s said that there are lies, damn lies, and statistics. Models are like statistics on steroids. Expect people who exaggerate facts for a living to abuse models. Distrust sensationalist media, politicians, salespeople, self-promoters, and anybody with an ax to grind.
This discussion is also pertinent to global climate models. A recent post on a professional message board illustrates a common concern. “Before making expensive policy decisions on the basis of climate models, I think it is reasonable that those models predict future temperatures accurately for a period of ten years. Twenty would be better” (citing Michael Crichton). We’ll see why global climate models are useful even though they will never be able to “predict” even ten years out. (How to make policy decisions is beyond the scope of this article.)
Understanding what modelers mean by prediction starts addressing these concerns. The skepticism illustrated above suggests a definition of “predict” that implies certainty. It’s consistent with synonyms like “foretell” and “prophesy.” These words conjure up an image of a seer who is able to see the future. The modeler’s understanding of “predict” is more nuanced. It is consistent with synonyms like “forecast.” A good modeling effort will seek out sources of uncertainty. This will illuminate the areas in which we lack understanding. In some cases, it will help us reduce uncertainty.
What, exactly, is a model? A model resembles something else in the real world of interest to us. This model acts like the real thing in most of the ways that are important. A model can never be a perfect representation of the real world. It will always be wrong in some ways. A small model of an airplane in a wind tunnel resembles its aerodynamic properties. A mechanical model can simulate the stresses on a machine part to know its capacity. A model can simulate a process such as patients entering and exiting a hospital. COVID-19 models simulate infection and death rates over time. Global climate models simulate temperatures, sea levels, and greenhouse gas concentrations over time. Both depend on assumptions about how people and institutions will behave. Models can represent both physical and human behavior. Physical models tend to be more accurate. Many of them use experimental data from prototypes. At the other extreme, economic models face perhaps the greatest challenges. They depend entirely on human behavior. Moreover, the people involved continually try to beat the models. COVID-19 and global climate models fall somewhere in the middle. Models can be numerical, qualitative, or even mental. Mental models are intuitive thought processes about some occurrence in the real world.
We’ll talk about models by way of a metaphor. We’ll use a mental model of a scenario that almost everyone is familiar with: cars and highway travel. The model of the car would include basic features such as the gas pedal, the brake, and the steering wheel. We can include details like fuel consumption, safety features, and rules of the road. This model, like all models, is incomplete and ignores many aspects of how a car operates. I can still use it to answer useful questions. Let’s use it to think about the trip my friend Jose is taking from San Diego, California, to Washington, D.C.
Models generate best estimates with varying levels of confidence. We can estimate that Jose, driving alone, will want to do about 500 miles a day, so his trip will take about 5 or 6 days. If I know Jose’s habits well, I can trust that he’ll follow through on his intention of arriving in D.C safely and quickly. In this case, I’ll feel more confident. I can tell our mutual friend Beto, “Expect Jose to arrive in D.C. in 5 or 6 days.” I’ll be less confident if I don’t know Jose as well or if I know he’s flexible and wants to see the sights. I’d have to tell Beto, “Maybe he’ll be there in under a week, or it could be a month if he decides to go to all the national parks. You know Jose.” If Beto or I claimed that Jose would be there in 5 or 6 days, we would misrepresent the uncertainty involved.
It’s the same for COVID-19 and global climate models. Models should report a range of outcomes with corresponding levels of confidence. They should also report assumptions made, so we can revise them as new information comes in. Good global climate and COVID-19 models do this well. These models don’t tell us certain outcomes. They produce a range of possible scenarios with caveats. Yet, models still help us understand how things work. We can understand better how events might play out. We can identify what we know and what we don’t. And when we learn new information, we can adjust our understanding.
Models explore possible limits for outcomes (we call this “bounding the problem”). I could use my mental model to figure that covering 2685 miles at an average speed of 80 mph would take about 33 hours. If Jose left today at 6 AM with a helper to drive nonstop, they could arrive as early as tomorrow at 6 PM with the time change. Is that really going to happen? I doubt it, but I’ve bounded the problem. I would be shocked if Jose arrived today at noon. It’s even faster than my fastest possible estimated time. If that occurred, I would have to reconsider my understanding of the situation. If I told Beto, “I ‘predict’ that Jose will be in D.C. tomorrow at 6 PM,” I would be a fool. If Beto thought that I had predicted a 6 PM arrival, he would be a fool.
Likewise, COVID-19 and global climate models can bound the problem. We can consider what could happen under “do-nothing” or “business as usual” scenarios. These scenarios are unlikely to occur. People will always adapt to a changing situation. Citizens will adopt some pandemic safety measures even without government intervention. Communities will adapt to changing climate in the absence of any government policy. Yet, these scenarios allow us to see how bad the pandemic might get, without thinking of it as a prediction.
Models identify dangerous courses of action to avoid. Let’s say that Jose calls me one day to tell me that he’s decided to set a speed record on a winding, two-lane, mountain road. We know that this is a bad idea, with possibly fatal consequences. Jose calls us the next day to announce that he didn’t set a record. He only spun out twice, but he’s OK and is continuing his trip. Was our model wrong? No, his actions were dangerous and remain so. What if Jose decided against it and took a safer road instead? Was our model wrong because he was never in as much danger as we said he was? No, our conclusion depended on a caveat (if you speed along the winding mountain road).
Similarly, both COVID-19 and global climate models can inform scenario planning. We can study what adverse outcomes might occur if we follow a particular course of action. In the pandemic, we prevented the worst-case by social distancing and closing businesses. Does that make our model wrong? On the contrary. The model was useful for avoiding the worst-case and knowing why we made the sacrifice. We should be happy that the model was informative enough to steer us clear of the worst outcome.
No model will capture all the important outcomes. This is because they simplify reality and must leave out some considerations. What if Jose got a call from his dear Aunt Greta in Boise (Jose is the son of a Salvadoran-Norwegian couple). Greta had an accident and needs his help. He detours to Boise and spends the next three weeks caring for his aunt. When he’s done, his business in D.C. is over. He returns to San Diego, never having arrived in D.C. Was our mental model wrong? Of course not. What happened is only an example that “shit happens.” An unexpected occurrence fell outside the assumptions of the model. This can always happen. Our mental model can still consider what will happen on Jose’s return trip from Boise to San Diego.
In the same way, COVID-19 and global climate models depend on what many people and institutions do. We have an excellent idea of how the physical part of the global climate system operates. We’re learning more every day about the novel coronavirus. The outcomes are still uncertain because they depend heavily on how people respond. The best we can do is make conservative assumptions, observe, and update our models. Maybe we’ll find a vaccine next month, rendering the pandemic all but over. If a model failed to “predict” this, would that make it a bad model? No. A miracle vaccine falls outside the model. A good model would still provide useful information such as how many people we need to inoculate. With this in mind, requiring global climate models to be right 10 to 20 years into the future is downright silly. No model will stand up to even ten years without encountering at least one big surprise. We don’t even know if poor aunt Greta will have an accident during Jose’s road trip. This rule is no better than saying, “I don’t care about your model.” Let’s be more reasonable. A model should fit the data well and make sense in hindsight once we know how people responded. If we have this, we can use the model to inform later decisions.
Closing thoughts: It’s said that “all models are wrong, but some are useful” (George Box). Models simplify reality. Our knowledge is imperfect. Real-world phenomena, especially those including people, are complex. Yet, if we keep their limitations in mind, models can be powerful tools to understand the world around us. Models inform our understanding of how a situation will likely unfold. We use models to plot out plausible scenarios and test limits. Models can make us honest about what we don’t know. When we gain more information, we refine our understanding and start again. Despite all this, models cannot predict with certainty. Too many of us want the “experts” to tell us “the answer” with certainty. This opens the door to gross misuses of models. Be careful with anybody who spins facts for a living. Be careful with your neighbor who only seems to want to win the argument. Be careful when you see yourself looking for a quick and easy answer. Dig a little bit deeper than the main conclusion. Consider how models can help improve your knowledge and intuition. This will make you more agile and flexible in facing the future.
Acknowledgment: Thank you to my colleagues Irfan Alvi, William Bulleit, Zachary Pirtle, and Jon Schmidt (in alphabetical order). Their insightful review and commentary on an earlier draft of this article helped clarify some important points.
