This article recalls some of the systematic errors made by the most visible epidemiologists, such as the belief that their model is truer than reality or the need to express pessimistic views to support forms of government through social acceptability. This discipline should not be allowed to have a monopoly on expertise for public decision-making.
By Laurent Mucchielli
On 6 March 2021, Le Monde published an interview with Simon Cauchemez that marks an important step in a form of logical absurdity that has been on display for almost a year now. When asked if he was wrong in announcing 300,000 to 350,000 deaths a year ago, the Scientific Council’s modeller answers « no », even though the estimated number of deaths due to Covid-19 is less than 90,000 to date (since not all the deaths with Covid-19 are due to Covid-19: INED still recommends waiting to conclude). Its central arguments are twofold: the measures taken are the cause of this massive gap, and the epidemic is not over.
This denial reveals a certain number of biases, widespread within the most official epidemiological community, including the one that consists of refusing to question anything. While this is particularly problematic when predictions are used to justify public policies, there are others that are regularly expressed in seminars and publications, such as the idea that public interventions should be pessimistic.
As a preamble to this text, it is worth noting the bad habit that has developed of using arguments of authority to include or exclude people who are ‘entitled’, in their capacity as epidemiologists, to criticise or even talk about modelling. This disciplinary defence is quite classic (it has been found in economics or finance for several decades). It remains shocking for all the interdisciplinary practitioners who defend, on the contrary, the idea that the formal model has the advantage of establishing a lingua franca that makes it debatable by anyone capable of reading it and asking relevant questions about it.
The President’s 400,000 deaths
What has characterised the period of crisis we are experiencing is an immoderate love of figures and numbers, produced and used in an often somewhat eccentric manner. The ‘400,000 deaths of the President of the Republic’, in his speech of 28 October, marks a turning point in the ability to mask reality with models.
After the first season of the epidemic, when at least 350,000 deaths had already been announced by the Ferguson team for England, which already created a significant discrepancy with the data, one could say that the models needed to be revised (we note in passing that S. Cauchemez worked for 8 years in the Ferguson laboratory, which may explain why he was chosen as the national expert, like his former colleague in England). A priori, however, they are based on the same principles: extrapolations of mortality or resuscitation from the data of the moment, and have not integrated the knowledge already acquired in September.
An important hypothesis emerged in June: there are humans that the virus cannot use to continue its journey between hosts. A significant proportion of the population would be protected from this virus even before encountering it – either the individual has « cross-immunity », or their mucous membranes do not allow the virus to penetrate – or they would not transmit it, even once infected. This would explain both the low rate of infection in and by children (to date Karl Friston gives the figure of 40% of children and 25% of adults who naturally will not be part of the chain of transmission), but also the fact that in many households only part of the family is infected if one is ill, or the limited spread of the virus on the Diamond Princess. Indeed, out of about 3,700 passengers exposed to the virus for the first time, only 700 caught it – while it is known that SARS-COV2 was circulating despite the cabin confinement, through the ventilation system (much less than without confinement, but it was still circulating). This is the idea that Michael Levitt (Nobel Prize in Chemistry) defended, and he was also able to show as early as March 2020 that the epidemic was not following an exponential curve, but that the growth rate was decreasing rapidly. These essential assumptions for making numerical predictions, which should at least be tested, were certainly not used to produce the figures, which were almost identical to those announced six months earlier. It is as if nothing had been learned.
For example, the question of the evolution of the virus (the « variants ») was publicly raised as early as September in France, and it was therefore understandable that several epidemics were already following each other, each building a more or less bell-shaped curve. But this fact was ignored in seminar discussions in France and abroad: as if a ‘second wave’, an automatic ‘rebound’ at the end of the containment period, had to be recognised, even though the curves of the models did not resemble the curves of the data at all. Thus, the « second wave », a modelling artefact, has become a truth for the general public, without it being possible to measure it in reality. The model is, once again, ‘truer’ than reality.
However, its truth value remained relative since Simon Cauchemez himself reduced the scale of the ‘announced catastrophe’: when the President said 400,000, he himself was betting on 300,000. It is worth noting the nonchalance with which 1/3 of the deaths were added to the scale. We can see that any hypothesis that is too pessimistic is always good to take among official epidemiologists. Especially since this figure poses a problem when it is dropped without correction in the public arena. Faced with an estimated lethality of 0.5% at the time, any citizen familiar with the rule of three knew that 400,000 deaths would only occur in a French population that was larger than it is at present (80 million inhabitants would be needed). Moreover, the age distribution of mortality was widely known and to come up with such a simply proportional figure showed that the models were not very ‘learned’. One wonders how much of the public lost confidence in the scientists’ claims in the face of this rather far-fetched extrapolation.
More seriously, this episode revealed that, while constantly waving figures and numbers around, the epidemiologists did not take the announced values seriously: the denial remains weak in the Check news, and no member of the scientific council publicly corrected the figure by insisting on the President’s exaggeration. Working in quantitative modelling and not defending the numerical value of the results: this is a lack of seriousness that really raises questions in the context of the concerns of the population.
Finally, and perhaps worst of all, at no time did the epidemiologists take care to specify how long all these people were expected to die. People do not worry in the same way when a disease will kill 300,000 people in five years (which seems possible if Covid-19 becomes endemic and if early care is still invisible in our country) or 300,000 deaths in one season. By playing on this ambiguity, the members of the Scientific Council have thus made politics rather than science (whose job is, on the contrary, to reduce any ambiguity).
This episode highlights not only the persistence in the error of method, the refusal to integrate new data, and – in the process – the lack of critical capacity of the major newspapers that relayed this information without particularly worrying about its validity.
World homogeneity and data aggregation
Other figures have appeared here and there, such as the famous number 6 for authorised private gatherings or tables in restaurants. One would be hard-pressed to know the model behind this norm, or even its logic: if gatherings of six people happen regularly, and if the people involved change groups regularly, propagation should happen about as well in groups of eight or four. Today, students are allowed to fill universities to 20%, without us even having a clue where such a figure could come from. Magic numbers regulate our lives.
As far as I know, few researchers have intervened to point out that the models used do not make it possible to deal with these questions of network modifications (and thus to « predict » the effect of confinement or other « distancing » rules). Gianluca Manzo pointed to the role of super-disseminators in the epidemic, which required serious attention to the reality of interactions. Without this reflection, how can we think of an effective policy since the heterogeneity of the number of links is essential in the dynamics of diffusion. I myself insisted on the fact that no firm prediction can be made with models of diffusion on networks, which are very much subject to the history of chance encounters (this effect is called « path dependence »).
In the absence of fine-grained analysis of these network issues, some epidemiologists have dealt with containment in the few models that represent them by homogeneously removing 70% of the interactions from each agent. It is obviously a low-level error to believe that this captures a credible representation of containment as it was experienced and, above all, to believe that this way of representing it has no influence on the results. It is not a question of reproaching colleagues for making these homogeneous models in order to observe the results and analyse them. The problem arises as soon as they are used as guides for action in a highly heterogeneous world.
Another absurd aspect of the official modelling was to produce national data extrapolation curves, without going through local situation analyses. However, during the first containment, the different zones observed were separate and the virus did not circulate between the regions any more than humans did: from the point of view of the spread of the virus, each of them was in a specific dynamic, and extrapolations of cases could only be made in an interesting and relevant way at the local level. However, one continued to hear assertions such as « my model is better for national than local data, so I’m using it nationally » – where any serious modeller would have concluded that the model should THEN go to the bin and be seriously revised. Strangely enough, colleagues did not seem to understand questions about local dynamics when asked in seminars, and never answered them. Finally, it was in November that Alexandra Henrion-Caude (in a video apparently deleted by YouTube) showed how informative an analysis of local dynamics would have been – especially in dealing with the onset of a second epidemic by knowing better the degree of naivety of the population and the epidemic forms to expect.
Model instead of reality
The idea of the ‘second wave’ has been discussed at length. Apparently, many respiratory specialists were concerned that this idea was spreading because they knew it was irrelevant. They expected to see a bell curve and then have to wait and see what would happen to the epidemic afterwards: would it return or not? In the models, however, if a containment doesn’t last very long, as soon as it is interrupted, you see an almost immediate rise in the number of cases. In our book, we show that this is true whatever the model. If this effect appears whatever the hypotheses, it is because it is mechanically linked to the SIR (Healthy, Infected, Recovered) modelling, not because it is « true ». In the infection figures, it was only with a time lag that we could see a rise in cases in July – not at all as the models would have ‘predicted’. We can then hypothesise that this result is linked to the fact that the dynamics of the virus itself are not taken into account in the models (its « seasonality »). The only way to believe in a « second wave » identical to that of the models is to deny once again the temporal and dynamic properties. It is worth noting that it was by focusing on this July upwelling, which they considered an anomaly, that the members of the Marseille IHU were able to detect a first variant, and warn the French with a good knowledge of the phenomenon as early as September 2020.
The question of data quality was regularly raised by demographers. For Hervé Le Bras, the limits of numerical analyses could be identified very early on. Others were able to recall the usual methods of collecting data to monitor an epidemic, or to show how official speeches were constantly using new measures to describe the epidemic, without any of them being well defined, nor their uncertainties specified – in particular with regard to declarations of deaths of people with Covid-19 or Covid-19.
Yet, despite the fundamental dependence on data of all models based on extrapolation over time, no attention was paid to this discussion by the most visible epidemiologists. The evidence sometimes swung from one evaluation model to another, and the quality of the measured data seemed ultimately incidental. For example, R0 is a proxy for a model based on infection data (not measured until there are enough tests), but is used as an input variable for many prediction models, without always taking precautions regarding the compatibility of assumptions, or the accumulation of uncertainties when embedding models.
Thus, the epidemiological satisfaction, complacently taken up by Libération, is based on the idea that neither short-term errors (in terms of occupancy of resuscitation beds a few weeks in advance) nor long-term errors (a whole year) are taken into account. Based on a model that was quickly validated on a few – imperfect – datasets, Simon Cauchemez claims to be certain that his predictions would have been realised had there been no confinement. We are talking about an order of magnitude difference of 1 to 3, which is enormous. The problem with this logic is that there is no counterfactual to show that he is right, since the models could not be validated. For example, the magnitude of the containment effect cannot be calculated with the type of model used, so no refutable predictions were made.
However, there are no observations of countries where the predicted disaster has been realised to the level expected, and indeed it is far from it. Reference can be made to a few worrying, highly localised situations, but these are exceptions rather than rules, and their causes should be analysed in detail.
Another kind of proof could indeed be provided by countries that have established the political counterfactual, such as Sweden. Unfortunately for our epidemiologists, it plays the role of reverse proof, with curves almost similar to ours, without any authoritarian rule having been imposed there, any more than the methodical destruction of the economy. If we are to say that « the models were not wrong » then we must explain why our mortality is similar to that of Sweden, which logically should have a death ratio about three times higher than ours. Several studies now show that strict containment has no discernible gain over lighter measures of distancing, that the obligation to stay at home does not guarantee a reduction in the circulation of the virus, point to the suppression of large gatherings as the best explanation to describe the history of the epidemic in Sweden (which also closed universities and high schools). However, it is often said that Sweden is « different » and cannot be compared to France. One can wonder on what criteria this assertion is based, since neither the characteristics of the population nor the organisation are for the moment explanatory of the mortality curves, and that Sweden obviously had a deficit of intensive care beds greater than ours, and a population density in the big cities equivalent to our metropolises.
Why do colleagues, and journalists, continue to repeat, often peremptorily, that their models are correct when it makes no sense without rigorous demonstration? The level of education being what it is, many Internet users visibly enjoy pointing out prediction errors, which further reduces confidence in the word of experts. Only in the land of Lewis Carroll does repetition make a proposition true (1).
Admitting some exaggerations, and explaining the changes in the models as time went by, would have challenged the reported tendency towards pessimism. Let’s do a thought experiment: it seems acceptable today to produce a 1 to 3 overestimate of the number of deaths, without the magnitude of the difference being discussed. Imagine, on the other hand, that someone had predicted 70,000 deaths rather than 90,000: the error is much smaller, the orders of magnitude are preserved. But this prediction would surely have earned the sender criticism, if he had even been listened to.
Finally, with hindsight, we can see that the supposedly quantitative models will have given us vague narratives, and will never have been convincingly validated by observations. Some end up being given as tools to generate raw ‘predictions’ thrown out to an uninformed public, with no clear qualification of the ‘precautions’ to be taken in interpreting them. The public is still at risk of realising that these models only predict ‘well’ on very rare occasions, which are themselves not very predictable. If this disappoints the public a little more, which has been governed for a year by arguments that it discovers to be so fragile, what will become of the authority of a « science » that is no longer discussed among scientists but is used above all for media promotion?
Interpretation of an intellectual shipwreck
What happened when the epidemic arrived in France and science was suddenly the focus of media and political attention? The same thing that our profession has known for years: the ANR launched a call for projects. Some projects were selected, and others were not: this created resentment and anxiety, and made everyone scramble even more to be visible or recognised – and thus ‘write papers’ quickly rather than collaborate to improve the collective understanding of problems. Although there were notable exceptions, initiated in March, such as the CoVprehension collective, where the explanation of phenomena was collectively analysed and written, the ModCov19 seminars (the official network launched by the CNRS around modelling) showed a very ‘usual’ world of scholarly exchange, where each person presents a paper produced in a small group, and where discussion is limited to a few minimal questions asked in ten minutes, without any possible criticism. This individualistic and discussion-shortening organisation is not a choice of the researchers themselves, it is simply the form that has become ritual, the emerging norm of the last twenty years. It is the result of a long slide linked to the culture of the project and individual evaluation: it is now better to make colleagues and their results or questions invisible than to confront their point of view and lose precious time in building a career or accessing funding.
Thus, although the Academy of Medicine indicated in July that wastewater analysis could effectively predict the presence of the virus in the population and anticipate hospital admissions two weeks in advance, some epidemiologists preferred to ignore this information and not cross-check it with their own forecasts in order to improve the models – even though the latter fulfilled exactly the same role. The appearance of the variants and their impact on the health strategy is only belatedly highlighted by the official epidemiology, whereas all this was clearly indicated by the IHU as early as September. Nor are the questions raised about the effectiveness of the vaccine, which have been going on for months, even mentioned as a limitation for the models presented.
This intense competition for access to resources is furthermore based on necessary implicit hierarchies, and the authority of certain bodies is vested in many scholarly discussions. For example: the results of the Institut Pasteur are still true, even if – if one believes their own analyses – some acknowledge that their models were not designed to incorporate the effect of containment, and that they had to improvise. However, the precariousness of researchers being perfectly installed, we realise that the team that has been giving indications to our government for months is composed almost exclusively of young non-statutory staff. The so-called ‘precarious’ researchers often have a little less experience and scientific culture because of their age, and therefore potentially a little less reflexivity, and a more limited network for discussing with their peers and testing their hypotheses and ideas extensively (because the relationship with peers is not reduced to ‘peer-review’ but is an exercise in daily exchange). Finally, it is known that it is very delicate for them to develop deep and risky research, or even to bring contradictions within the research spaces on which their career depends.
This passive acceptance of competition goes hand in hand with a lack of knowledge of disciplines with which epidemiologists should be familiar. For example, how can we understand that some researchers in the hard sciences confuse democracy with respect for their recommendations? Because they do not have a culture of decision support that integrates science, they are not familiar with the idea of arbitration between possible options as developed by economists, for example, nor do they know that a decision in a complex universe can never be based on a single analytical criterion. For a year, it was a trade-off between perceived acceptability in the population and the number of deaths – and « acceptability » is a concept of social manipulation rather than democratic discussion. By confusing their personal ideas with the common good, the scientists who claimed to be able to help govern the country demonstrated their lack of knowledge of philosophy, epistemology, decision support, or even the sociology of science. This bias is unfortunately widespread, and one cannot imagine solving it without profoundly transforming educational paths.
It is to be hoped that this rapid exposure of the structural flaws in the work of the scientific community and expertise, of which only a few elements are given here, will allow for an intelligent review of science policy trajectories in the coming years. Especially if emerging infectious diseases are expected in large numbers, we cannot afford such a lack of discussion among scientists. Unbridled competition has never been the way to go if knowledge is to be of use to all.
(1) « The right place for the Snark! I’ve told you twice:
That should be enough to encourage you.
The right place for the Snark! I’ve told you three times:
What I say three times is absolutely true.
Lewis Carroll, The Snark Hunt
Juliette ROUCHIER, research director at the CNRS in economics and environment, specialist in the use of agent models applied to social sciences, and former head of the GDR « Policy Analytics » (Innovative decision support for public policies).