Shiny marketing presentations about Cochrane's successes are usually based on misleading quantitative figures and the obsession with greater economic turnover but they fall short dearly on delivering the scientific and democratic quality that is actually the only way of delivering better health-care information, synthesis and evidence to help people to take more informed decisions. Especially in health-care, more does not mean better. The Cochrane leadership´s approach reflects a head-in-the-sand “follow the money” approach that ignores most of the key strategic concerns of many of its key members and the wider health-care community.
Membership numbers and balance sheets do not reflect Cochrane´s existential crisis over transparency, openness and democratic accountability. The fact that calls for change by a large majority of centre directors in Europe and Latin America and almost half of the previous Governing Board members have fallen on deaf ears and open denial on the part of the central leadership is unprecedented in any democratic civil society organization. On a qualitative level many of the most highly qualified, experienced and recognised medical researchers around the world are increasingly critical in varying degrees of the organizational and scientific direction Cochrane has taken in the last number of years. (From Moynihan to Goldacre, from Iaonnidis to Paul Glasziou, from Godlee to Jefferson, from Dickerson to Rivaud, from Erviti to Bonfill etc.) If one wants to be quantitative too, have a look at the 10000 signatures and 2000 personal messages in support of Peter Gøtzsche (https://www.ipetitions.com/petition/letter-to-danish-minister-of-health-against#facebook-share), who is considered by John Iaonnidis “one of the greatest scientists of our times”. His expulsion has sent a very clear message to the health-care community that Cochrane does not tolerate strong critics of the pharmaceutical industry and the medical establishment.
The repression of dissent and open scientific debate has had a devastating impact on the public and professional trust in Cochrane. As reflected on these lists of support for Gøtzsche and many other public declarations in articles and in social media the present Cochrane leadership is intellectually and professionally isolated. The London Cochrane leadership are more and more criticised among researchers and professionals who are in favour of change in our biomedical innovation and knowledge models. This qualitative fact underlines a sort of “Cochrexit” where the London central office has cut itself off from the world. It is a given that they will maintain control of the Board but this can be a pyrrhic victory in light of the disrepute the crisis has caused.
The lack of internal democracy is scandalous and openly recognized by most. As I describe in my no gracias blog, most Board members are not encouraged nor allowed to be active in decision-making processes and they have become a final stage rubber-stamp of central executive team decisions and any critical strategic discussion is strongly discouraged within a Board agenda ambience where 95% of the issues considered by the Board are related to the publishing business model, “the brand” and financial progress. Cochrane is not “inclusive and diverse” because it is intolerant of diversity and criticism, persecuting and rejecting open democratic debates and contested elections. The expulsion of Peter Gøtzsche without any democratic process by a margin of 6-5, the censoring of a “voting guide” by leading Cochrane members for the recent elections and the totally opaque, incomprehensible candidate information as described by Ben Goldacre on Twitter, are just a few examples.
The assertion that strong centralization policies are in favour of “quality” is absurd in a context in which the central office actively rejects and even considers “disloyal” any open debate on the scientific weaknesses of Cochrane´s present business model that often ignores the biased, manipulated and opaque nature of industry-sponsored or influenced journal articles as the sole basis for many systematic reviews. Greater central control is not improving quality because it chooses to ignore most of the underlying structural issues that determine the quality of the systematic reviews.. The core business of Cochrane are its Systematic Reviews yet in the last decade Cochrane has dragged its heels in response to insistent concerns that they are largely synthesized information from industry-sponsored studies. Cochrane has done little to address the increasing need for reviews to be undertaken using accessible regulatory data (Clinical Study Reports and patient-level clinical data). In contrast, by giving total priority to its publishing business model the Cochrane London team has refused any honest open dialogue within the Board or other organs about what the basis of “trusted evidence” actually is. A few of the “costs” of the maintaining a large, expensive London staff is a highly criticized policy of a lack of open data to back up reviews and the lack of responsiveness for immediate open access to reviews demanded by many EU member states and international research organizations. More Knowledge Translation and language translations will not advance “better evidence” if they are based on weak, redacted or inaccessible clinical trial evidence and biased methodological premises. Cochrane reviews can even end up amplifying and certifying pharma industry marketing spins. What many long-time Cochrane members are calling for is not the production of more reviews but fewer ones of greater scientific quality, credibility and independence. But, of course, greatly slowing down the “systematic review assembly line production”(even called “chain reviews”) would reduce the revenue of the central office which is the overriding objective of the present Cochrane leadership. And you can´t deal with quality of content later! And quality of content is not mainly a technical issue but one of strategic scientific and moral choices.
It is very unfortunate that a significant portion of Cochrane editors have financial conflicts of interest with promoters of the treatments they are evaluating. This undermines trust in the recommendations. Up to half of the review authors are allowed to have conflicts of interest with the products they are considering.
Open access statistics (increasing by 1%) given are misleading or mean very little because after one year of moratorium behind a paywall almost all publications become open access anyway. The key economic question is how Cochrane will respond to the demand of immediate open access in the future and how this will affect the core of its business model.
The idea that Cochrane national leaders are key advocates of evidence-based medicine for Cochrane flies in the face of the fact that most of these leaders from Germany to Austria to Finland to Spain to Denmark to most of Latin America are calling for democratic change in the way Cochrane works. See the German letter from the EBM network: https://www.ebm-netzwerk.de/pdf/stellungnahmen/letter-cochrane-20181004.pdf.
All the data about the increase of the quantity of online downloads and visits says very little about the influence of Cochrane reviews on decision-making and health-care policy. These are qualitative questions that need another kind of metrics. In fact, the ongoing public crisis of Cochrane (unless there is a strong reaction) will probably have a very negative impact on that influence.
The fact is that the core business of the Cochrane central office is not providing “trusted evidence” but producing its “products” in the Cochrane Library which is principally the Cochrane Database of Systematic Reviews. Most of the work is done by Cochrane review groups around the world while the revenues are almost exclusively controlled by the London office and rarely shared with regional centres that have their own independent funding. Centre director complaints over policies that affect their national scientific work and income are routinely brushed off and ignored.
This has little or nothing to do with improving quality and a lot to do with top-down control. It is false, the assertion: “We are financially sustainable as an organization at central and group levels”. At a central level it is certainly true, but the groups are often struggling with a relatively high turnover and financial instability.
For example, when one well-known author of some of the most influential and impacting Cochrane reviews ever published, on flu vaccines, asked for financial support to update his reviews, he was laughed off by the chief editor. Cochrane researchers serve the central office in exchange for the Cochrane stamp and some editing services, but they should not expect any financial support from substantial and growing income of the central office. Inversely, the central office uses its resources to try to control and centralize the work of the centres into a “unified message and brand,” often in contradiction with the needs of the network.
The problem with this core business model, aside from squashing the essential open scientific model needed for more trusted and efficient results, is that the present Cochrane leadership ignores that many of the systematic reviews can be flawed, faulty and incomplete “products” because they are based on journal articles with pervasive publication bias that often amplifies benefits and minimizes possible harms of a medical treatment.
“Quality” is often exaggerated by Cochrane because it is a non-brainer that most reviews of pharmaceuticals are likely to be biased or incomplete as they are based on industry sponsored publications usually without access to the original clinical data and much less data independent of industry sources. There is no point in improving the technical processes, increasing the number of publications, promoting more “knowledge translation” and language translations if the overwhelming source of evidence, usually journal articles, is contaminated by bias, selective reporting, lack of trial transparency and conflicts of interest. It is noteworthy, the almost total absence of Cochrane advocacy on these issues of transparency in the EU, at the WHO and before other institutions. This silence is not a coincidence nor is it due to a lack of resources. It reflects, as declared by Cochrane´s CEO, the need of aligning any advocacy in accordance with the needs of Cochrane´s “products,” not public health needs.
Aside from carrying out systematic reviews, a core activity of Cochrane has been the development of methodology. Many of the key intellectual powerful forces that have led the development of these methodologies such as EQUATOR and others have distanced themselves from Cochrane due to the lack of productive dialogue with the present leadership. This exodus of top methodologists will weaken Cochrane's position as methodological leader but also affect SRs which will be seen as less trustworthy.
For drugs, implantables and biologics, such as vaccines, we know or suspect that trial publications are affected by reporting bias. As Tom Jefferson has stated: “The result is garbage in garbage out with a seal of approval: The Cochrane logo.” This issue is never considered by the Cochrane leadership because it threatens their publishing business model based on reviews of journal articles. Business has trumped science in Cochrane. This is a win-lose situation for decision makers.
The Cochrane leadership chooses to ignore in its methodology, public promotion and almost total absence of critical policy advocacy the structural problems of the “raw material” used for systematic reviews: the data/articles produced, rationed and manipulated by the pharmaceutical industry. It is no coincidence that Peter Gøtzsche and thousands of researchers around the world who support him are precisely those who have publicly pointed out these fundamental problems with our medical innovation system. Without recognizing and trying to correct these limitations and biases at every stage, Cochrane´s work loses credibility and trust. The Cochrane leadership has generally ignored civil society opinion makers and scientific leaders who have led campaigns for trial transparency, against conflicts of interest, in favour of open data and for new medical innovation models not based on patent monopolies.
As many observers have noted, there are generally two confronting paradigms about the future of Cochrane. One is a collaborative based on open science principles that is not afraid of publicly questioning some of the basic social, economic and scientific premises of our current medical research model dominated by big pharma and the other is a much more centralized, functionalist, conformist and conservative approach that prioritizes the current scientific publishing model that precludes any important distancing from pharmaceutical industry interests. In the end it is a question of moral choice.
David Hammerstein
