A Synthesis of Evidence for Policy from Behavioural Science during COVID-19

"[S]urvey data strongly supported how critical it is for policy to understand collective behaviour, shared values and effects on marginalized populations to be effective at minimizing harms during a pandemic."

Scientific evidence - including evidence from the behavioural and social sciences - plays an important role in policy decisions. One example of science impacting policy comes from an April 2020 review produced by 42 academics from 8 countries and multiple academic disciplines that included a list of 19 behavioural insights ("claims") deemed most relevant to the COVID-19 pandemic. (See Related Summaries, below.) In a 2-year period, this paper was cited over 3,000 times, and governments around the world formulated pandemic policy strategies explicitly on the basis of the behavioural concepts highlighted in the paper. Its recommendations covered topics such as official messaging on social distancing, how to get a vaccine once they were available, and the need to work within communities to create real impact. This follow-up paper assesses 747 pandemic-related research articles that empirically investigated those 19 claims. It reports the scale of evidence and whether evidence supports them to indicate applicability for policymaking. One goal is to present a pragmatic, expert-driven method for evaluating and synthesising evidence that is suitable for informing public policy (both related to COVID-19 and future applications), making those assessments public in a way that promotes transparency and builds trust with the public.

Two independent teams, involving 72 reviewers from more than 30 countries, took part in the exercise. Participants included both original authors and an independent team of evaluators to select and assess evidence relevant to these claims, all of whom were blinded to names and assessments. They assessed the 19 behavioural policy recommendations by evaluating available articles based on the level of evidence they include. Ratings range from purely opinion or theory to large-scale, replicated field studies, as well as the size and direction of effects reported. Their evaluations primarily focused on the scale and scope of empirical findings directly related to the claims, although the compiled data also highlights methods, geographical settings, and specific behaviours. They then synthesised the evidence within each claim to formulate a summary evaluation.

Overall, the review indicates that the 2020 article generally identified highly relevant topics of study in the pandemic and, to an extent, the direction of associated findings. In particular, it identified (i) relevant behaviours during the pandemic (both positive and negative), (ii) likely barriers to mitigating the spread of the disease, and (iii) major social challenges that would be faced by policymakers.

Specifically, the study finds evidence for 18 of 19 claims in the 2020 paper, including those related to sense of identity and community connectedness, leadership and trust, public health messaging, social cohesion, and misinformation. Of the 18, the 2020 paper correctly identified 16 relevant behavioural concepts during the pandemic, as well as likely barriers to mitigating spread of the disease and social challenges that would be faced by policymakers. The strongest evidence supported claims that anticipated that culture, polarisation, and misinformation would be associated with policy effectiveness. Claims suggesting that trusted leaders and positive social norms increased adherence to behavioural interventions also had strong empirical support, as did appeals to social consensus or bipartisan agreement. Broadly, consistent with original expectations, evidence has supported the value of highlighting the cooperation of other people to promote health behaviours.

Importantly, no effects were in the opposite direction from the original predictions - that is, no recommendations from the 2020 paper led to a consistent backfire effect. That said, the researchers found no effect for two proposed policies related to effective public messaging (that messages should emphasise benefits to the recipient, and that they should focus on protecting others). Notably, the team found no evidence to review for one high-profile recommendation in the 2020 paper, which suggested the phrasing "physical distancing" is preferable to "social distancing".

The study reveals that public health interventions that received the most attention were not necessarily the ones best supported by the most evidence. For example, handwashing was widely promoted as a strategy for stopping the spread of COVID-19, yet study effects were small to null, particularly compared to masking, isolation, distancing, and vaccines.

Analysis of 463 papers containing data showed generally large samples; 418 involved human participants with a mean of 16,848 (median of 1,699). That statistical power underscored improved suitability of behavioural science research for informing policy decisions.

The 2023 study also identifies several domains missing from the 2020 paper. These included threat and risk perception, nudging, stress and coping, the role of inequality and racism, skepticism toward science, incentivising behaviours beyond simply describing benefits (e.g., by providing financial rewards for vaccination), and the absence of clear leadership. Another theme not discussed was how traditional forms of mass media might have undermined the potentially helpful role of descriptive norms by giving disproportionate attention to anti-vaccination, conspiracy, and other beliefs that did not reflect expert or even majority opinion in the general public. Efforts of academics and public health officials may be thwarted if media policies around "equal coverage" are implemented in ways that amplify false norms and harmful, fringe ideas, given how easily it is to manipulate or control narratives that are not rooted in evidence.

Finally, the paper provides recommendations to help researchers and policymakers respond to future pandemics and disasters, such as the need to: study global populations, do more field testing, and be more specific in formulating testable questions. Also, considering the myriad challenges to linking scientific research to real-world practice that are amplified during pandemics, the researchers propose forging alliances. For example, academics could proactively engage with potential end-users of scientific evidence (e.g., organisations delivering public services) to find out where and what input they would value. In turn, policymakers are urged to open lines of communications with academics who research their professional area. Building such relationships "enables us to develop a robust and relevant evidence base and to ensure that we marshal our collective energies and resources so that we are able to use science to best effect in the service of serving, protecting, promoting and prioritizing the well-being of populations."

Full list of authors, with institutional affiliations: Kai Ruggeri, Columbia University Mailman School of Public Health, University of Cambridge, and US Air Force/New York Air National Guard; Friederike Stock, Max Planck Institute for Human Development and Humboldt University of Berlin; S. Alexander Haslam, University of Queensland; Valerio Capraro, University of Milan-Bicocca; Paulo Boggio, Mackenzie Presbyterian University; National Institute of Science and Technology on Social and Affective Neuroscience - CNPq; Naomi Ellemers, Utrecht University; Aleksandra Cichocka, University of Kent; Karen M. Douglas, University of Kent; David G. Rand, Massachusetts Institute of Technology; Sander van der Linden, University of Cambridge; Mina Cikara, Harvard University; Eli J. Finkel, Northwestern University; James N. Druckman, Northwestern University; Michael J. A. Wohl, Carleton University; Richard E. Petty, Ohio State University; Joshua A. Tucker, New York University; Azim Shariff, University of British Columbia; Michele Gelfand, Stanford University; Dominic Packer, Lehigh University; Jolanda Jetten, University of Queensland; Paul A. M. Van Lange, Vrije Universiteit Amsterdam and University of Cologne; Gordon Pennycook, Cornell University; Ellen Peters, University of Oregon; Katherine Baicker, University of Chicago; Alia Crum, Stanford University; Kim A. Weeden, Cornell University; Lucy Napper, Lehigh University; Nassim Tabri, Carleton University; Jamil Zaki, Stanford University; Linda Skitka, University of Illinois Chicago; Shinobu Kitayama, University of Michigan; Dean Mobbs, California Institute of Technology; Cass R. Sunstein, Harvard University; Sarah Ashcroft-Jones, Columbia University Mailman School of Public Health and University of Oxford; Anna Louise Todsen, University of Oxford; Ali Hajian, University of Tehran; Sanne Verra, Utrecht University; Vanessa Buehler, Cowry Consulting; Maja Friedemann, University of Oxford; Marlene Hecht, Max Planck Institute for Human Development and Humboldt University of Berlin; Rayyan S. Mobarak, University of Maryland; Ralitsa Karakasheva, Junior Researcher Programme; Markus R. Tünte, University of Vienna; Siu Kit Yeung, The Chinese University of Hong Kong; R. Shayna Rosenbaum, York University and Baycrest Academy for Research and Education; Žan Lep, University of Ljubljana and Educational Research Institute; Yuki Yamada, Kyushu University; Sa-kiera Tiarra Jolynn Hudson, University of California Berkeley; Lucía Macchia, City, University of London; Irina Soboleva, Duke Kunshan University; Eugen Dimant, University of Pennsylvania and CESifo; Sandra J. Geiger, University of Vienna; Hannes Jarke, University of Cambridge; Tobias Wingen, University of Bonn; Jana B. Berkessel, University of Mannheim; Silvana Mareva, University of Cambridge and University of Exeter; Lucy McGill, University College Dublin and University of Groningen; Francesca Papa, Organisation for Economic Co-operation and Development; Bojana Većkalov, University of Amsterdam; Zeina Afif, The World Bank; Eike K. Buabang, Trinity College Dublin; Marna Landman, University of Pretoria; Felice Tavera, University of Cologne; Jack L. Andrews, University of Oxford and University College; Aslı Bursalıoğlu, Loyola University Chicago; Zorana Zupan, University of Belgrade; Lisa Wagner, University of Zurich; Joaquín Navajas, Universidad Torcuato Di Tella and Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET); Marek Vranka, Charles University; David Kasdan, Sungkyunkwan University; Patricia Chen, University of Texas at Austin and National University of Singapore; Kathleen R. Hudson, University of Illinois Chicago; Lindsay M. Novak, University of Illinois Chicago; Paul Teas, University of Illinois Chicago; Nikolay R. Rachev, Sofia University St. Kliment Ohridski; Matteo M. Galizzi, London School of Economics; Katherine L. Milkman, The Wharton School of the University of Pennsylvania; Marija Petrović, University of Belgrade; Jay J. Van Bavel, New York University; Robb Willer, Stanford University