Expansion of cognitive testing for off-field concussion screening in elite rugby players: A cohort study



      Current off-field concussion screening instruments have sub-optimal accuracy and additional testing domains may be necessary to detect the full spectrum of concussion presentations. This study aimed to determine if additional cognitive tests add utility to off-field screening for sport-related concussion.


      Reproducibility and diagnostic accuracy cohort studies were performed in the 2017 and 2018 seasons of the Super Rugby competition, conducted in Argentina, Australia, Japan, New Zealand, and South Africa.


      Abridged versions of Stroop (score, time), Spatial Memory (score, failed trials), and Trail Making Trial-B (time, errors) cognitive tests, modified for off-field use, were examined. Players performed baseline testing prior to each season. Cases undergoing off-field screening as part of the World Rugby Head Injury Assessment Process underwent evaluation with the same cognitive tests during competition matches. Agreement between repeated pre-season baseline measurements, and the diagnostic accuracy of off-field testing against a clinical reference standard of concussion, was evaluated.


      Data were available for repeated preseason baseline testing in 644 players, and 100 cases undergoing off-field concussion screening. There was little individual agreement across pre-season baseline assessments for all tests (Lin's correlation and Gwets AC1 coefficients ranging between 0.2 and 0.3). There was significantly worse performance for the time taken to complete the modified Stroop Test in concussed players undergoing off-field screening, compared to non-concussed players (median time 21.1 v 18.4 s, p < 0.01; area under the receiver operating characteristic curve 0.7 (95% CI 0.5–0.8)). Other cognitive measures did not discriminate between injured and un-injured players, with no-statistically significant differences in distribution medians (p = 0.6–0.9) and AUROC values close to 0.5.


      The time taken to perform a modified Stroop Test may provide additional diagnostic accuracy if added to current off-field concussion screening tools. Abridged Spatial Memory and Trail Making Trial-B tests did not discriminate between concussed and non-concussed players.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Science and Medicine in Sport
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • McCrea H.J.
        • Perrine K.
        • Niogi S.
        • et al.
        Concussion in sports.
        Sports Health. 2013; 5: 160-164
        • Jordan B.D.
        The clinical spectrum of sport-related traumatic brain injury.
        Nat Rev Neurol. 2013; 9: 222-230
        • McCrory P.
        • Meeuwisse W.H.
        • Aubry M.
        • et al.
        Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012.
        Br J Sports Med. 2013; 47: 250-258
        • Fuller G.W.
        • Tucker R.
        • Starling L.
        • et al.
        The performance of the World Rugby Head Injury Assessment Screening Tool: a diagnostic accuracy study.
        Sports Med Open. 2020; 6: 2
        • Fuller C.W.
        • Fuller G.W.
        • Kemp S.P.
        • et al.
        Evaluation of World Rugby’s concussion management process: results from Rugby World Cup 2015.
        Br J Sports Med. 2017; 51: 64-69
        • Lezak M.D.
        Neuropsychological Assessment.
        Oxford University Press, New York2012
        • Sherman E.M.S.
        • Spreen O.
        • Strauss E.
        Compendium of Neuropsychological Tests: Administration, Norms, and Commentary.
        Oxford University Press, New York2006
        • Ben-David B.M.
        • Nguyen L.L.
        • van Lieshout P.H.
        Stroop effects in persons with traumatic brain injury: selective attention, speed of processing, or color-naming? A meta-analysis.
        J Int Neuropsychol Soc. 2011; 17: 354-363
        • Lange R.T.
        • Iverson G.L.
        • Zakrzewski M.J.
        • et al.
        Interpreting the trail making test following traumatic brain injury: comparison of traditional time scores and derived indices.
        J Clin Exp Neuropsychol. 2005; 27: 897-906
        • Woods D.L.
        • Wyma J.M.
        • Herron T.J.
        • et al.
        The effects of repeat testing, malingering, and traumatic brain injury on computerized measures of visuospatial memory span.
        Front Hum Neurosci. 2015; 9: 690
        • Patricios J.
        • Fuller G.W.
        • Ellenbogen R.
        • et al.
        What are the critical elements of sideline screening that can be used to establish the diagnosis of concussion? A systematic review.
        Br J Sports Med. 2017; 51: 888-894
        • Leiter R.
        Partington's pathway test.
        Psychol Serv Cent Bull. 1949; 1: 9-20
        • Salthouse T.A.
        What cognitive abilities are involved in trail-making performance?.
        Intelligence. 2011; 39: 222-232
        • Scarpina F.
        • Tagini S.
        The Stroop Color and Word Test.
        Front Psychol. 2017; 8
        • Stroop J.R.
        Studies of interference in serial verbal reactions.
        J Exp Psychol Gen. 1992; 121: 15-23
        • Corsi P.M.
        Human Memory and the Medial Temporal Region of the Brain.
        McGill Univ, Montreal1973
        • Kessels R.P.
        • van Zandvoort M.J.
        • Postma A.
        • et al.
        The Corsi Block-Tapping Task: standardization and normative data.
        Appl Neuropsychol. 2000; 7: 252-258
        • Raftery M.
        • Kemp S.
        • Patricios J.
        • et al.
        It is time to give concussion an operational definition: a 3-step process to diagnose (or rule out) concussion within 48 h of injury: World Rugby guideline.
        Br J Sports Med. 2016; 50: 642-643
        • Echemendia R.J.
        • Meeuwisse W.
        • McCrory P.
        • et al.
        The Sport Concussion Assessment Tool 5th Edition (SCAT5): background and rationale.
        Br J Sports Med. 2017; 51: 848-850
        • Collie A.
        • Maruff P.
        • Makdissi M.
        • et al.
        CogSport: reliability and correlation with conventional cognitive tests used in postconcussion medical evaluations.
        Clin J Sport Med. 2003; 13: 28-32
        • Fukui T.
        • Sugita K.
        • Sato Y.
        • et al.
        Cognitive functions in subjects with incidental cerebral hyperintensities.
        Eur Neurol. 1994; 34: 272-276
        • Hashimoto R.
        • Meguro K.
        • Lee E.
        • et al.
        Effect of age and education on the Trail Making Test and determination of normative data for Japanese elderly people: the Tajiri Project.
        Psychiatry Clin Neurosci. 2006; 60: 422-428
        • Armengol C.G.
        Stroop Test in Spanish: Children’s norms.
        Clin Neuropsychol. 2002; 16: 67-80
        • Heaton R.K.
        • Grant I.
        • Matthews C.G.
        Comprehensive Norms for an Expanded Halstead-Reitan Battery: Demographic Corrections, Research Findings, and Clinical Applications; With a Supplement for the Wechsler Adult Intelligence Scale-Revised (WAIS-R).
        Psychological Assessment Resources, 1991
        • Brooks A.
        • MacDonald K.
        • Wasylyk N.
        • et al.
        Normative values for the SCAT3 in adolescent athletes.
        Br J Sports Med. 2014; 48: 573-574
        • Fuller G.W.
        • Govind O.
        • Tucker R.
        • et al.
        Sport concussion assessment tool-third edition normative reference values for professional Rugby Union players.
        J Sci Med Sport. 2018; 21: 347-351
        • Buderer N.M.
        Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity.
        Acad Emerg Med. 1996; 3: 895-900
        • Bigler E.D.
        Neuropathology of Mild Traumatic Brain Injury: Correlation to Neurocognitive and Neurobehavioral Findings Brain Neurotrauma: Molecular, Neuropsychological, and Rehabilitation Aspects.
        Taylor & Francis Group, LLC, Boca Raton FL2015
        • Bohnen N.
        • Jolles J.
        • Twijnstra A.
        Modification of the stroop color word test improves differentiation between patients with mild head injury and matched controls.
        Clin Neuropsychol. 1992; 6: 178-184
        • Koss E.
        • Ober B.A.
        • Delis D.C.
        • et al.
        The Stroop color-word test: indicator of dementia severity.
        Int J Neurosci. 1984; 24: 53-61
        • Sisco S.M.
        • Slonena E.
        • Okun M.S.
        • et al.
        Parkinson’s disease and the Stroop color word test: processing speed and interference algorithms.
        Clin Neuropsychol. 2016; 30: 1104-1117
        • Lansbergen M.M.
        • Kenemans J.L.
        • van Engeland H.
        Stroop interference and attention-deficit/hyperactivity disorder: a review and meta-analysis.
        Neuropsychology. 2007; 21: 251-262
        • Yengo-Kahn A.M.
        • Hale A.T.
        • Zalneraitis B.H.
        • et al.
        The Sport Concussion Assessment Tool: a systematic review.
        Neurosurg Focus. 2016; 40E6
        • Soukup V.M.
        • Ingram F.
        • Grady J.J.
        • et al.
        Trail Making Test: issues in normative data selection.
        Appl Neuropsychol. 1998; 5: 65-73
        • Perianez J.A.
        • Rios-Lago M.
        • Rodriguez-Sanchez J.M.
        • et al.
        Trail Making Test in traumatic brain injury, schizophrenia, and normal ageing: sample comparisons and normative data.
        Arch Clin Neuropsychol. 2007; 22: 433-447