Maths models may give policy makers major headache

Mathematical models that predict policy-driving scenarios - such as how a new pandemic might spread - may be too complex and delivering ‘wrong’ answers, a new study reveals.

Experts are using increasingly detailed models to better predict phenomena or gain more accurate insights in a range of key areas, such as environmental/climate sciences, hydrology and epidemiology.

But the pursuit of complex models as tools to produce more accurate projections and predictions may not deliver because more complex models tend to produce more uncertain estimates.

Researchers from the Universities of Reading, Birmingham, Princeton, Barcelona and Bergen published their findings today in Science Advances. They reveal that expanding models without checking how extra detail adds uncertainty limits the models’ usefulness as tools to inform policy decisions in the real world.

Samuele Lo Piano, Postdoctoral Research Fellow on Flexibility in Energy Demand at the University of Reading, said: “There has been a tendency in research over the past few decades to escalate model complexity thanks to the higher computational power available.

“This means academics are able to seek higher accuracy in the output, and hence a better representation of the system modelled, but their models may fail because adding more information actually results in increasing the uncertainty in the output.

“Greater model complexity eventually most likely leads to predictions with a higher level of uncertainty, rather than more accuracy.”

“Relevant examples of this trend are global hydrological models, which started as simple models and now aim at representing the global water cycle, or the European project Destination Earth, or the modelling of Covid-19 spreading elaborated at Imperial College London in 2020.”

This tendency to produce more inaccurate results affects any model without training or validation data used to check its output’s accuracy - affecting all global models such as those focused on climate-change, hydrology, food-production, and epidemiology, as well as models projecting estimates into the future, regardless of the scientific field.

Researchers recommend that the drive to produce increasingly detailed mathematical models as a means to get sharper estimates should be reassessed.

Arnald Puy, Associate Professor in Social and Environmental Uncertainties at the University of Birmingham, said: “We suggest that modelers should calculate the model’s effective dimensions (the number of influential parameters and their highest-order interaction) before making the model more complex. This allows to check how the addition of model complexity affects the uncertainty in the output. Such information is especially valuable for models aiming to play a role in policy making.

“Both modelers and policy makers benefit from understanding any uncertainty generated when a model is upgraded with novel mechanisms.

“Modelers tend not to submit their models to uncertainty and sensitivity analysis but keep on adding detail. Not many scholars are interested running such an analysis on their model if it risks showing that the emperor runs naked and its alleged sharp estimates are just a mirage.”

Excess complexity prevents scholars and public alike to ponder the appropriateness of the models’ assumptions, often highly questionable. Puy and his team note, for example, that global hydrological models assume that irrigation optimises crop production and water use - a premise at odds with practices of traditional irrigators.