Drug and medical device companies have a long history of selective reporting the results of clinical trials used to test their new products and even of burying results they don’t like. In an effort to make it harder for companies to manipulate reporting of results, the federal government (NIH) created ClinicalTrials.gov, where companies and others could register their clinical trial protocols when the study started. Trial sponsors are also supposed to report any changes to the protocol and analysis plan as well as the final results once the trial is done. In addition, most major medical and public health journals require that a clinical trial be registered with ClinicalTrials.gov to be considered for publication.
In addition, to improve and standardize reporting results in peer reviewed journals, medical journal editors developed the CONSORT (Consolidated Standards of Reporting Trials) “an evidence-based, minimum set of recommendations for reporting randomized trials. It offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.” Specifically. “The CONSORT Statement comprises a 25-item checklist and a flow diagram. The checklist items focus on reporting how the trial was designed, analyzed, and interpreted; the flow diagram displays the progress of all participants through the trial.”
The combined goal of these two systems is to ensure that trials are completely reported in a standardized way without spinning or burying negative or other unpleasant results.
While not developed with the tobacco industry in mind, these standards are particularly important for tobacco because the industry has a long history of spinning results by manipulating the protocol and statistical analysis (such Philip Morris’ study on additives) or by simply not publishing unfavorable results (such as RJ Reynolds’ negative study on snus for smoking cessation).
In light of this history, Nicholas DeVito and colleagues searched ClinicalTrials. gov for Juul Labs trials that appeared to be covered the Food and Drug Administration Amendments Act 2007 (FDAAA) that requires registration and disclosure of trials relevant to FDA regulation. Their paper, “E-cigarette manufacturers’ compliance with clinical trial reporting expectations: a case series of registered trials by Juul Labs” published in Tobacco Control found that none of the 5 clinical trials they found fully reported the results (table above).
In a news story about the paper in BMJ,
A Juul spokesperson told The BMJ that as part of the premarket tobacco product application process the company provided the Food and Drug Administration (FDA) with the raw data, clinical results, and statistical code underlying the analyses of all five of the clinical studies detailed in the article. “While the FDA is ultimately responsible for analysing the scientific evidence about our products and making a determination regarding their public health impact, we also appreciate that scientists and the public health community have an important interest in better understanding our research,” they said.
There are several problems with this argument:
- It puts the burden on the FDA to plow through the data and do a forensic analysis of the statistical approach that Juul took. (This is particularly problematic given the huge number of applications for new tobacco product that the FDA is faced with evaluating.)
- Journals and meeting abstracts rarely provide enough space for all the details related to a study, whereas ClinicalTrials.gov has a standardized reporting system and no length limits on the reports.
- Where the results are published can be important, such as when Juul purchased an entire issue of American Journal of Health Behavior, to publish its results, raising questions about the nature of the “peer review.”
Juul also made these arguments in a comment on the DeVito paper on the PubPeer website, which has been a popular outlet of e-cigarette advocates. Among other things, Juul argued that the FDAAA did not obligate them to register their trails and report results on ClinicalTrials.gov, but they did so voluntarily.
DeVito and colleagues offered a detailed point-by-point response to Juul’s arguments, which ended by saying:
Regardless of the exact regulatory status of these trials, a matter we hope the FDA will offer further clarity on, we strongly encourage Juul Labs to consider amending their existing registrations with the results of these trials. ClinicalTrials.gov offers a rapid, public, open source repository for trial results that enhances their availability, transparency, and discoverability and should not preclude subsequent peer reviewed publications. It is clear from their response that Juul Labs recognises the importance of trial registration and reporting to the public and scientific community, and we welcome this recognition. ClinicalTrials.gov allows for full results to be reported, without space limitations or delays associated with journal publication, and with a clear public link to all prespecified outcomes and other registered details. While the FDAAA compels this type of reporting, there is no limitation on voluntary submission of results if trials are not covered under the law. We strongly believe it is in the public interest that all outcomes for all trials that form PMTA [Premarket Tobacco Product Applications, to get authority to sell the products in the USA] applications should be fully and publicly reported in a timely manner. Currently this does not appear to be the case even if it may have improved since our evaluation.
Whether compelled to report by legal requirements or not, we hope Juul will consider setting a strong precedent for transparency in tobacco-industry e-cigarette research by reporting their full trial results for these, and future, trials to ClinicalTrials.gov and assuage any doubts about selective reporting of results or outcomes.
These are particularly important points because the FDA keeps PMTA applications secret, which makes it virtually impossible for the public, including outside experts to comment on the evidence that companies submit to FDA in their efforts to get tobacco products authorized. FDA could also allow outside comment by referring important PMTA applications to its Tobacco Products Scientific Advisory Committee, something it has not done to date.
In light of the problems DeVito and colleagues identified, FDA should do two things:
- Issue a policy guidance unequivocally stating than any clinical trial data submitted as part of a tobacco product application must have been completely reported on ClinicalTrials.gov before being submitted to FDA.
- Refer the Juul and any other important PMTA applications to the Tobacco Products Scientific Advisory Committee so that these issues can recieve a full public airing before awarding any PMTA authorizations for any e-cigarettes.
Here is the abstract for the DeVito paper:
Background: Electronic cigarettes (e-cigarettes) are a frequently debated topic in public health. It is essential that clinical trials examining e-cigarettes are fully and accurately reported, especially given long-standing concerns about tobacco industry research. We assess the reporting of clinical trials sponsored by Juul Labs, the largest e-cigarette company in the USA, against accepted reporting standards.
Methods: We searched ClinicalTrials.gov for all trials sponsored by Juul Labs and determined those with registry data consistent with coverage by the Food and Drug Administration (FDA) Amendments Act 2007 (FDAAA). For trials with a primary completion date more than 1 year earlier, we searched ClinicalTrials.gov, the academic literature and a Juul-funded research database (JLI Science) for results. For located results, we compared reported outcomes with registered outcomes in line with Consolidated Standards of Reporting Trials (CONSORT) reporting guidelines.
Results: We located five registered trials sponsored by Juul Labs that appeared covered by the FDAAA 2007 in the public data. All five trials did not have results available on ClinicalTrials.gov. We found one publication and four poster presentations reporting results for four of the five covered trials outside of ClinicalTrials.gov. Of 61 specified outcomes, 28 were CONSORT compliant. Specific outcome reporting issues are detailed.
Discussion: Our findings raise substantial concerns regarding these trials. Clinicians, public health professionals, and the public cannot make informed choices about the benefits or hazards of e-cigarettes if the results of clinical trials are not completely and transparently reported. Clarification and potential enforcement of reporting laws may be required.
The full citation is: DeVito NJ, Drysdale H, McKee M, Goldacre B. E-cigarette manufacturers’ compliance with clinical trial reporting expectations: a case series of registered trials by Juul Labs. Tob Control. 2021 Jun 14:tobaccocontrol-2020-056221. doi: 10.1136/tobaccocontrol-2020-056221. Epub ahead of print. PMID: 34127550. It is available here.
Stanton Glantz blog 