Together with Dr Michael Ganslmeier (University of Exeter), we have co-authored a new study published in the Proceedings of the National Academy of Sciences (PNAS), titled Estimating the Extent and Sources of Model Uncertainty in Political Science. The article addresses a fundamental challenge in empirical social science: the extent to which published findings depend on defensible but ultimately variable modelling decisions.

Assessing model uncertainty is crucial to quantitative political science. Yet, most available sensitivity analyses focus only on a few modeling choices, most notably the covariate space, while neglecting to jointly consider several equally important modeling choices simultaneously. In this article, we combine the exhaustive and systematic method of the Extreme Bounds Analysis with the more multidimensional logic underpinning the multiverse approach to develop an approach to sensitivity analyses. This allows us to systematically assess the degree and sources of model uncertainty across multiple dimensions, including the control set, fixed effect structures, standard error types, sample selection, and dependent variable operationalization.

We then apply this method to four prominent topics in political science: democratization, institutional trust, public good provision, and welfare state generosity. Results from over 3.6 bn estimates reveal widespread model uncertainty, not just in terms of the statistical significance of the effects, but also their direction, with most independent variables yielding a substantive share of statistically significant positive and negative coefficients depending on model specification. We compare the strengths and weaknesses of three distinct approaches to estimating the relative importance of different model specification choices: nearest 1-neighbor; logistic; and deep learning. All three approaches reveal that the impact of the covariate space is relatively modest compared to the impact of sample selection and dependent variable operationalization. We conclude that model uncertainty stems more from sampling and measurement than conditioning and discuss the methodological implications for how to assess model uncertainty in the social sciences.