Poor quality statistical reporting in the biomedical literature is not uncommon. Here is another example by Cirio et al. (2016). The study itself is well planed, executed and reported. The aim was to assess whether heated and humidified high flow gases delivered through nasal cannula (HFNC) improve exercise performance in severe chronic obstructive pulmonary disease (COPD) patients. It all started when I saw their Fig.1. Here is my attempt to reproduce it
In total there are 12 patients tested twice; once under the control test and once under the HFNC test. The outcome of interest is the endurance time (Tlim; y axis). This is practically how long each test lasted. The authors hypothesized that HFNC would improve exercise performance, that is the test would last longer. This was the case since Tlim increased for all subjects under the HFNC test (see figure 1). Moreover, this increase reached statistical significance (p-value = 0.015) - ready to publish! Looking at the plot I was pondered about the “outlying” patient (red dot). His/her Tlim increased by a whooping 400 seconds! This is huge compared to the other patients. Then I wondered how would the results change if we excluded him/her from the analysis? And here is where the problems start.
There is no way from the text to figure out which test was used to produce the p-value of 0.015. Is is a paired t-test or a Wilcoxon test? (they mention both in the statistical analysis section). So it is impossible to evaluate and/or try to reproduce the results.
Having abandoned the idea of being able to reproduce the analysis I started thinking about reporting guidelines, hence the title of this post. I thought the journal must have guidelines for reporting statistical analysis. No, it does not and unfortunately, most of the biomedical journals don’t have such guidelines even though 40 years ago O’Fallon and colleges recommended that “Standards governing the content and format of statistical aspects should be developed to guide authors in the preparation of manuscripts” (O’Fallon et al. 1978). Since then many have repeated the message. A few sporadic attempts are usually editorials such as Cummings and Rivara (2003), Curran-Everett and Benos (2004) and Arifin et al. (2016).
Recently, Lang and Altman (2013) published a comprehensive set of statistical reporting guidelines suitable for medical journals - the SAMPL guidelines. “The SAMPL guidelines are designed to be included in a journal’s Instructions for Authors”. So the journals just need to refer to them! As there are many general reporting guidelines based on the study design as such CONSORT, STROBE, PRISMA etc (see http://www.equator-network.org/) that authors in many journals must adhere to, I believe the SAMPL guidelines is a big step forward on reporting statistics. The only journal (that I know of) that suggests the use of the SAMPL guidelines is the British Journal of Dermatology (Hollestein and Nijsten 2015). (I’ll keep adding to this list).
Now that the guidelines exist, let’s make use of them.
Arifin, Wan Nor, Abdullah Sarimah, Bachok Norsa’adah, Yaacob Najib Majdi, Ab Hamid Siti-Azrin, Musa Kamarul Imran, Abd Aziz Aniza, and Lin Naing. 2016. “Reporting Statistical Results in Medical Journals.” The Malaysian Journal of Medical Sciences: MJMS 23 (5). School of Medical Sciences, Universiti Sains Malaysia: 1.
Cirio, Serena, Manuela Piran, Michele Vitacca, Giancarlo Piaggi, Piero Ceriana, Matteo Prazzoli, Mara Paneroni, and Annalisa Carlucci. 2016. “Effects of Heated and Humidified High Flow Gases During High-Intensity Constant-Load Exercise on Severe Copd Patients with Ventilatory Limitation.” Respiratory Medicine 118. Elsevier: 128–32.
Cummings, Peter, and Frederick P Rivara. 2003. “Reporting Statistical Information in Medical Journal Articles.” Archives of Pediatrics & Adolescent Medicine 157 (4). American Medical Association: 321–24.
Curran-Everett, Douglas, and Dale J Benos. 2004. “Guidelines for Reporting Statistics in Journals Published by the American Physiological Society.” Am Physiological Soc.
Hollestein, LM, and Tamar Nijsten. 2015. “Guidelines for Statistical Reporting in the British Journal of Dermatology.” British Journal of Dermatology 173 (1). Wiley Online Library: 3–5.
Lang, Thomas A, and Douglas G Altman. 2013. “Basic Statistical Reporting for Articles Published in Biomedical Journals: The ‘Statistical Analyses and Methods in the Published Literature’ or the Sampl Guidelines”.” Handbook, European Association of Science Editors 256. Wiley Online Library: 256.
O’Fallon, JR, SD Dubey, DS Salsburg, JH Edmonson, A Soffer, and T Colton. 1978. “Should There Be Statistical Guidelines for Medical Research Papers?” Biometrics. JSTOR, 687–95.