Determining when a scientific discipline is ready to inform policy is knotty. To add to this complexity, criteria for determining whether the knowledge is robust enough vary depending on the issues. Nonetheless, a London School of Economics study found that more and more governments are turning to science to inform public policy. In times of crisis, the need for insight is exceptionally pressing. Yet evidence-based policymaking is even more difficult when there is little data and little time to gather it.
During the COVID-19 pandemic, the scientific community was called upon to contribute to the rapid understanding and treatment of the crisis, at the risk of doing hasty work. Faced with this tremendous challenge, what pitfalls did they fall into? Under what conditions can research usefully inform public debate and help political decision-making?
Is research less rigorous under pressure?
In a fascinating article on the shortcomings of research acceleration (available on WPRN), Christopher Grieser notes several disturbing facts. A meta-analysis of nearly 1,700 clinical trials on COVID-19 shows that only 5% of them demonstrated compliance with measures ensuring that other factors could not explain a stated causal relationship. Among these, only one-fifth of the 700 were randomized controlled trials, employed concealments of subject assignment. The COVID-19 trials samples were also smaller than usual to allow for rapid publication... Samples are often too small to reliably detect significant effects.
Grieser also points out that, during the pandemic, 18% of life scientists switched their research subject to something related to COVID-19. He stresses that this unintended redundancy wastes resources and that this ‘migration’ of scientists may have affected the quality of research. Indeed, since newcomers have little or no experience in coronavirus research, the quality of their work is often lower than that of specialists. According to his sources, while in February 2020 most pre-publications on coronaviruses were valuable contributions, by the end of May 2020 only one in 50 pre-publications was really strong. Thus, attempts to increase research speed sometimes prove counterproductive: false leads and a less robust basis generate waste of researchers’ time. More broadly, Grieser’s theory is that the enormous public attention on COVID-19 may have slowed down coronavirus research through “inappropriate redirection of research efforts”.
Rushed to assess
While scientists have always spent time reviewing their peers’ work, the hours now spent doing so are unprecedented, according to Grieser's research. During the pandemic, the scientific community devoted much energy to refuting low-quality research to prevent it from becoming misinformation. A low-quality paper used to be ignored or reviewed by no more than two researchers; now, several dozen researchers might review it. This came as an emergency response to prevent policymakers or the media from picking up on erroneous or misleading pre-publications again, as they did with hydroxychloroquine (which derailed the search for a COVID-19 treatment with hundreds of clinical trials on hydroxychloroquine because of political interest, despite preliminary inconclusive results). Other options exist to manage this risk while freeing up valuable time for more useful research activities.
Separating the wheat from the chaff
One of the reasons why the media and politicians have brough pre-publications into the public debate - often omitting to sort out their quality - is that science is based on an highly lengthy peer review process. While a political decision with huge consequences may have to be taken in a few days or weeks, the review process takes months. According to a British study on the management of scientific knowledge in the event of a crisis, this intrusion of outside eyes on work in progress seems inevitable. The paper suggests creating a flag system to differentiate each reviewing stage, from submitting an article to final publication. This would avoid all documents being treated as equally valid.
In the context of a pandemic, to reduce publication delays, the British researchers also recommends drawing inspiration from the model of medical journals where publication decisions are a binary “Yes/No”. Following the example of the Lancet’s ‘rapid evidence review’ on the psychological impact of quarantine, they suggest checking that the work is methodologically sound (not that the reviewer would have done it that way), not wasting time on rewriting to improve the style, and not soliciting additional experience that ‘might’ prove interesting.
Expert... in one’s field
Another change to be made is to resist the temptation to speak out of one’s field of expertise, despite journalists’ requests. After analyzing more than 800 articles written by researchers in the humanities and social sciences in the media between March and July 2020 (study to be found on the WPRN), Elsa Bansard and Anne-Coralie Bonnaire of the CNRS conclude that «researchers often lean outside their disciplinary area to give an opinion that is undoubtedly enlightened, but not well-founded in terms of their field of research. (...) For example, in Le Figaro's newspaper, a lawyer speaks about education and a sociologist about psychology. In the newspaper Le Monde, historians have a say about crisis management policy. The English research mentioned above highlights the need to create experts databases to enable the media to identify qualified people quickly. Current information networks (e.g., university and academic societies databases) do not appear to be up to the task. The gap between the general public and scientists mustn't widen further.
The pitfalls of the political instrumentalization of science
Shall science give an output for evidence-based policy-making? The COVID-19 pandemic made it a heated debate. Justifying decisions with research is not without difficulties and risks for policy makers. An article published by the University of Cambridge highlights the UK government’s lack of rigor in its use of behavioral science. In particular, it recalls the concern of six hundred behavioral scientists who sent an open letter to Downing Street questioning the scientific basis for the decision not to adopt a strict policy of social distancing. In this article, Anne-Lise Sibony points out that better-documented phenomena than “behavioral fatigue” could have been used to support the decision, but that this would not have made the decision unquestionable. A rigorous scientific approach recognizes that the probability and distribution of behavioral phenomena are unknown in this COVID-19 crisis and that some specific mechanisms may also mitigate these phenomena.
Is scientific advice possible?
For Sibony, the British case is an example of how governments can misuse behavioral concepts and tarnish the reputation of research-based policies. But even if the government had respected scientific rigor, could behavioral knowledge have informed its policy? Is it possible for sufficient scientific consensus to emerge in a setting with so many unknowns? In a comparative study of anti-Covid policies in Jordan and Sweden registered on WPRN, Élise Tancoigne and Marianne Noël show that opposite strategies were chosen based on behavioral science. In both countries, experts in the discipline have criticized the decisions taken.
Scientific debate at the service of democracy?
Following their research on the role of the humanities and social sciences in times of crisis (to be found on the WPRN database), Maryse Bresson and Pierre Guibentif from MSH Saclay point out that the erosion of the democratic model is partly fuelled by an excessive distancing between the scientific and social worlds. They recommend “facilitating an active reappropriation of science by non-scientists”. In their study referenced on the WPRN, researchers from the London School of Economics (LSE) also suggest that further efforts should be made to explain scientific concepts used by politicians and to clarify the distinctions between different schools of thought within disciplines.
While better communication to the general public is necessary to maintain or rebuild trust in research, the LSE study shows that public disagreement among experts does not necessarily undermine the credibility of science. The MSH Saclay researchers point out that this is excellent news because the scientific activity is, above all, an exercise of debate. According to them, the discussion initiated and fuelled by scientists on collective issues should guide political decisions. It is not the scientific discourse itself that should do it. From this perspective, as Maryse Bresson and Pierre Guibentif explain, humanities could help revitalize politics: "The modern sciences took off before the development of democratic political regimes and they provided these regimes, particularly through academies, with models for organizing collective action through consultation and for debating ideas and arguments. (...) If science has played a role in the formation of modern democracies, its role may well be essential in the current efforts to rehabilitate democracy."
Appendix
This memorandum is based on resources from the World Pandemic Research Network.
Maryse Bresson, Pierre Guibentif, Elsa Bansard and Anne-Coralie Bonnaire, «The Public Role of Social Sciences and Humanities in Times of Pandemics https://wprn.org/item/459452
Humanities and social sciences help bridge the gap between science, politics, economics, and people. This vocation requires participation in public debate. A team from MSH Paris-Saclay examined the current capacity of social sciences to play this role effectively via, among other methods, a quantitative and qualitative analysis of public statements made by social scientists in the French media during the first wave of COVID-19 (more than 800 articles written by humanities and social sciences researchers between March and July 2020).
Christopher Grieser, «Accelerated Research. Theorizing the Speed of Scientific Knowledge Production on Covid-19» - Discussion Paper https://wprn.org/item/417752
This study conducted at the Technical University of Berlin explores why attempts to accelerate research are inherently perilous. Using a comprehensive causal model, four main dimensions are explored: the speed of individual research activities, the relevance of knowledge contributions, the speed of knowledge flow, and the number of parallel research activities.
Jet Sanders, Emma Watson, Sandra Obradovic, Liam Delaney, Alessia Tosi, «Trust in emerging science: The study of perceptions and acceptability of behavioural science in the UK Covid-19 policy response» https://wprn.org/item/458152
This research by a London School of Economics team explores feelings about key actors and concepts in behavioral science in the UK. Using a mixed methodological design, it maps the media and public discourse (newspaper and Twitter discussions) surrounding behavioral science contributions from March 2020 over 24 weeks.
Anne-Lise Sibony, «The UK COVID-19 Response: A Behavioural Irony?» in Alberto Alemanno, «European Journal of Risk Regulation Special Issue: Taming COVID-19 by Regulation»https://wprn.org/item/445552 The contributions collected in this special issue of the European Journal of Risk Regulation from the University of Cambridge provide a first analysis of the “surprisingly uncoordinated, sometimes unscientific response to an essentially predictable event in a geopolitically fragmented world.” Anne-Lise Sibony’s article focuses on the case of the UK government’s use of behavioral science.
Marianne Noël and Elise Tancoigne, «Performing expertise in times of pandemics
https://wprn.org/item/434452
In their comparative analysis of Jordan and Sweden, Elise Tancoigne and Marianne Noël (University of Geneva) show that these two states use scientific expertise for similar purposes, which leads them to ask: when and why are such performances still credible for the public of each country, and under what historical and cultural conditions?