Variability in the analysis of a single neuroimaging dataset by many teams - Inserm - Institut national de la santé et de la recherche médicale
Journal Articles Nature Year : 2020

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer (1, 2) , Felix Holzmeister (3) , Colin F Camerer (4) , Anna Dreber (5, 3) , Juergen Huber (3) , Magnus Johannesson (5) , Michael Kirchler (3) , Roni Iwanir (1) , Jeanette A Mumford (6) , R. Alison Adcock (7) , Paolo Avesani (8, 9) , Blazej M Baczkowski (10) , Aahana Bajracharya (11) , Leah Bakst (12) , Sheryl Ball (13) , Marco Barilari (14) , Nadège Bault (15) , Derek Beaton (16) , Julia Beitner (17, 18) , Roland G Benoit (10) , Ruud M.W.J. Berkers (10) , Jamil P Bhanji (19) , Bharat B Biswal (20, 21) , Sebastian Bobadilla-Suarez (22) , Tiago Bortolini (23) , Katherine L Bottenhorn (24) , Alexander Bowring (25) , Senne Braem (26, 27) , Hayley R Brooks (28) , Emily G Brudner (19) , Cristian B Calderon (27) , Julia A Camilleri (29, 30) , Jaime J Castrellon (7) , Luca Cecchetti (31) , Edna C Cieslik (29, 30) , Zachary J Cole (32) , Olivier Collignon (9, 14) , Robert W Cox (33) , William A Cunningham (34) , Stefan Czoschke (18) , Kamalaker Dadi (35) , Charles P Davis (36, 37) , Alberto De Luca (38) , Mauricio R Delgado (19) , Lysia Demetriou (39, 25) , Jeffrey B Dennison (40) , Xin Di (20, 21) , Erin W Dickie (34) , Ekaterina Dobryakova (41) , Claire L Donnat (42) , Juergen Dukart (30, 29) , Niall W Duncan (43, 44) , Joke Durnez (42) , Amr Eed (45) , Simon B. Eickhoff (30, 29) , Andrew Erhart (28) , Laura Fontanesi (46) , G. Matthew Fricke (47) , Shiguang Fu (48) , Adriana Galván (49) , Remi Gau (50) , Sarah Genon (30, 29) , Tristan Glatard (51) , Enrico Glerean (52) , Jelle J Goeman (53) , Sergej A.E. Golowin (43) , Carlos González-García (27) , Krzysztof J Gorgolewski (42) , Cheryl L Grady (16) , Mikella A Green (7) , João F Guassi Moreira (49) , Olivia Guest (22) , Shabnam Hakimi (7) , J. Paul Hamilton (54) , Roeland Hancock (37) , Giacomo Handjaras (31) , Bronson B Harry (55) , Colin Hawco (56) , Peer Herholz (57) , Gabrielle Herman (56) , Stephan Heunis (58, 59) , Felix Hoffstaedter (30, 29) , Jeremy Hogeveen (47) , Susan Holmes (42) , Chuan-Peng Hu (60) , Scott A Huettel (7) , Matthew E Hughes (61) , Vittorio Iacovella (9) , Alexandru D Iordan (62) , Peder M Isager (58) , Ayse I Isik (63) , Andrew Jahn (62) , Matthew R Johnson (32) , Tom Johnstone (61) , Michael J.E. Joseph (56) , Anthony C Juliano (41) , Joseph W Kable (64) , Michalis Kassinopoulos (57) , Cemal Koba (31) , Xiang-Zhen Kong (65) , Timothy R Koscik (66) , Nuri Erkut Kucukboyaci (67, 41) , Brice A Kuhl (68) , Sebastian Kupek (3) , Angela R Laird (24) , Claus Lamm (69) , Robert Langner (30, 29) , Nina Lauharatanahirun (64) , Hongmi Lee (70) , Sangil Lee (64) , Alexander Leemans (38) , Andrea Leo (31) , Elise Lesage (27) , Flora Li (71) , Monica Y.C. Li (37, 72) , Phui Cheng Lim (32) , Evan N Lintz (32) , Schuyler W Liphardt (47) , Annabel B Losecaat Vermeer (69) , Bradley C Love (22, 73) , Michael L Mack (34) , Norberto Malpica (74) , Theo Marins (23) , Camille Maumet (75) , Kelsey Mcdonald (7) , Joseph T Mcguire (12) , Helena Melero (74) , Adriana S Méndez Leal (49) , Benjamin Meyer (60) , Kristin N Meyer (76) , Glad Mihai (10, 77) , Georgios D. Mitsis (57) , Jorge Moll (23, 42) , Dylan M Nielson (33) , Gustav Nilsonne (78, 79) , Michael P Notter (80) , Emanuele Olivetti (8, 9) , Adrian I Onicas (31) , Paolo Papale (31, 81) , Kaustubh R Patil (30, 29) , Jonathan E Peelle (11) , Alexandre Pérez (57) , Doris Pischedda (82, 83, 84) , Jean-Baptiste Poline (57, 85) , Yanina Prystauka (37) , Shruti Ray (20) , Patricia A Reuter-Lorenz (62) , Richard C Reynolds (33) , Emiliano Ricciardi (31) , Jenny R Rieck (16) , Anais M Rodriguez-Thompson (76) , Anthony Romyn (34) , Taylor Salo (24) , Gregory R Samanez-Larkin (7) , Emilio Sanz-Morales (74) , Margaret L Schlichting (34) , Douglas H Schultz (32) , Qiang Shen (48) , Margaret A Sheridan (76) , Jennifer A Silvers (49) , Kenny Skagerlund (54) , Alec Smith (13) , David V Smith (40) , Peter Sokol-Hessner (28) , Simon R Steinkamp (30) , Sarah M Tashjian (49) , Bertrand Thirion (35) , John N Thorp (86) , Gustav Tinghög (54) , Loreen Tisdall (42, 46) , Steven H Tompson (87) , Claudio Toro-Serey (12) , Juan Jesus Torre Tresols (35) , Leonardo Tozzi (88) , Vuong Truong (43, 44) , Luca Turella (9) , Anna E van ‘t Veer (89) , Tom Verguts (27) , Jean M Vettel (90, 64) , Sagana Vijayarajah (34) , Khoi Vo (7) , Matthew B Wall (39, 22) , Wouter D Weeda (89) , Susanne Weis (30, 29) , David J White (61) , David Wisniewski (27) , Alba Xifra-Porxas (57) , Emily A Yearling (37) , Sangsuk Yoon (91) , Rui Yuan (88) , Kenneth S.L. Yuen (60) , Lei Zhang (69) , Xu Zhang (37) , Joshua E Zosky (32) , Thomas E. Nichols (25) , Russell A Poldrack (42) , Tom Schonberg (1)
1 TAU - Tel Aviv University
2 Dartmouth College [Hanover]
3 Leopold Franzens Universität Innsbruck - University of Innsbruck
4 CALTECH - California Institute of Technology
5 SSE - Stockholm School of Economics
6 University of Wisconsin-Madison
7 Duke University [Durham]
8 FBK - Fondazione Bruno Kessler [Trento, Italy]
9 UNITN - Università degli Studi di Trento = University of Trento
10 IMPNSC - Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig]
11 WUSTL - Washington University in Saint Louis
12 BU - Boston University [Boston]
13 Virginia Tech [Blacksburg]
14 UCL - Université Catholique de Louvain = Catholic University of Louvain
15 Plymouth University
16 RRI - Rotman Research Institute at the Baycrest Centre
17 UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam
18 Goethe University Frankfurt
19 Rutgers University [Newark]
20 NJIT - New Jersey Institute of Technology [Newark]
21 UESTC - University of Electronic Science and Technology of China [Chengdu]
22 UCL - University College of London [London]
23 D'Or Institute for Research and Education [Rio de Janeiro]
24 FIU - Florida International University [Miami]
25 University of Oxford
26 VUB - Vrije Universiteit Brussel [Bruxelles]
27 UGENT - Universiteit Gent = Ghent University = Université de Gand
28 University of Denver
29 Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf]
30 Jülich Research Centre
31 IMT - School for Advanced Studies Lucca
32 University of Nebraska–Lincoln
33 NIH - National Institutes of Health [Bethesda, MD, USA]
34 University of Toronto
35 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
36 Lancashire Teaching Hospital
37 UCONN - University of Connecticut
38 UMCU - University Medical Center [Utrecht]
39 Imperial College London
40 Temple University [Philadelphia]
41 Kessler Foundation [East Hanover]
42 Stanford University
43 Taipei Medical University
44 TMU-ShuangHo Hospital [New Taipei City]
45 Instituto de Neurociencias de Alicante
46 Unibas - Université de Bâle = University of Basel = Basel Universität
47 The University of New Mexico [Albuquerque]
48 Zhejiang University of Technology
49 UCLA - University of California [Los Angeles]
50 UCL - Université catholique de Lille
51 Concordia University [Montreal]
52 Aalto University
53 LUMC - Leiden University Medical Center
54 LIU - Linköping University
55 Western Sydney University
56 CAMH - Centre for Addiction and Mental Health [Toronto]
57 McGill University = Université McGill [Montréal, Canada]
58 TU/e - Eindhoven University of Technology [Eindhoven]
59 Epilepsy Centre Kempenhaeghe [Heeze]
60 LIR - Leibniz-Institut für Resilienzforschung [Mainz]
61 Swinburne University of Technology (Hawthorn campus)
62 University of Michigan [Ann Arbor]
63 Max Planck Institute for Empirical Aesthetics [Frankfurt am Main]
64 University of Pennsylvania
65 Max Planck Institute for Psycholinguistics
66 Carver College of Medicine, University of Iowa
67 NJMS - Rutgers New Jersey Medical School
68 University of Oregon [Eugene]
69 Universität Wien = University of Vienna
70 JHU - Johns Hopkins University
71 NJU - Nanjing University
72 Haskins Laboratories
73 The Alan Turing Institute
74 URJC - Universidad Rey Juan Carlos = Rey Juan Carlos University
75 Empenn - Neuroimagerie: méthodes et applications
76 UNC - University of North Carolina [Chapel Hill]
77 TU Dresden - Technische Universität Dresden = Dresden University of Technology
78 Karolinska Institutet [Stockholm]
79 Stockholm University
80 UNIL - Université de Lausanne = University of Lausanne
81 NIN - Netherlands Institute for Neuroscience
82 Charité - UniversitätsMedizin = Berlin University Medicine
83 HU Berlin - Humboldt-Universität zu Berlin = Humboldt University of Berlin = Université Humboldt de Berlin
84 TUB - Technical University of Berlin / Technische Universität Berlin
85 UC Berkeley - University of California [Berkeley]
86 Columbia University [New York]
87 Human Research and Engineering Directorate [Aberdeen Proving Ground]
88 Stanford School of Medicine [Stanford]
89 Universiteit Leiden = Leiden University
90 UC Santa Barbara - University of California [Santa Barbara]
91 University of Dayton
Magnus Johannesson
Roni Iwanir
  • Function : Author
Stefan Czoschke
Joke Durnez
  • Function : Author
Remi Gau
Tristan Glatard
  • Function : Author
  • PersonId : 867504
Enrico Glerean
  • Function : Author
Jelle J Goeman
  • Function : Author
  • PersonId : 987058
Susan Holmes
  • Function : Author
  • PersonId : 943876
Claus Lamm
Sangil Lee
  • Function : Author
  • PersonId : 1000034
Flora Li
  • Function : Author
Michael L Mack
  • Function : Author
  • PersonId : 892724
Camille Maumet
Benjamin Meyer
  • Function : Author
  • PersonId : 782319
  • IdRef : 200081810
Gustav Tinghög
Khoi Vo
  • Function : Author
Sangsuk Yoon
Lei Zhang
  • Function : Author
  • PersonId : 1011221
Xu Zhang
  • Function : Author
  • PersonId : 1012368
Thomas E. Nichols
  • Function : Correspondent author
Russell A Poldrack
  • Function : Correspondent author
Tom Schonberg

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Domains

Neuroscience

Dates and versions

inserm-02914443 , version 1 (11-08-2020)

Identifiers

Cite

Rotem Botvinik-Nezer, Felix Holzmeister, Colin F Camerer, Anna Dreber, Juergen Huber, et al.. Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 2020, 582 (7810), pp.84-88. ⟨10.1038/s41586-020-2314-9⟩. ⟨inserm-02914443⟩
463 View
0 Download

Altmetric

Share

More