Validation of Korean Version of the Oxford Cognitive Screen (K-OCS), a Post Stroke-Specific Cognitive Screening Tool

Article information

Ann Rehabil Med. 2025;49(1):5-14

Publication date (electronic) : 2025 February 13

doi : https://doi.org/10.5535/arm.240099

Eunyoung Cho, PhD¹, Sungwon Choi, PhD², Nele Demeyere, PhD³, Rina Kim, MD⁴, Ikhyun Lim, MD¹, MinYoung Kim, MD, PhD^,1

¹Department of Rehabilitation Medicine, CHA Bundang Medical Center, CHA University School of Medicine, Seongnam, Korea

²Department of Psychology, Duksung Women’s University, Seoul, Korea

³Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, England

⁴Department of Psychiatry, Brookdale University Hospital and Medical Center, Brooklyn, NY, United States

Correspondence: MinYoung Kim Department of Rehabilitation Medicine, CHA Bundang Medical Center, CHA University School of Medicine, 59 Yatap-ro, Bundang-gu, Seongnam 13496, Korea. Tel: +82-31-780-1872 Fax: +82-31-780-3449 E-mail: kmin@cha.ac.kr

Received 2024 November 5; Revised 2025 January 9; Accepted 2025 January 10.

Abstract

GRAPHICAL ABSTRACT

INTRODUCTION

More than 70% of the patients with stroke experience post-stroke cognitive impairment (PSCI), depending on the type of stroke, definition, and time point of assessment [1,2], and 32% of the survivors experience persistent cognitive decline up to 3 years after stroke [3]. Assessment of potential PSCI is important, as the phenomenon may be reversible in the early phase and presence of co-morbid conditions may affect cognition [4]. Numerous international clinical treatment guidelines for stroke recommend that survivors be routinely assessed for cognitive impairment using cognitive screening tools, followed by a comprehensive assessment according to the results [5,6]. In terms of measuring cognitive impairment after stroke, improvements in the screening tools are required to reflect better insights into the disease pathophysiology and symptoms [7]. The Mini-Mental State Examination (MMSE) [8] and Montreal Cognitive Assessment (MoCA) [9] have been developed to screen for dementia in general and are currently being used to evaluate post-stroke cognitive function, but their application in the field of stroke care faces numerous challenges. As a representative example, difficulties may emerge after lesions in the brain language domain owing to limitations in the expressive or receptive language abilities of the patients with stroke [10,11]. In practical terms, approximately 20% of stroke patients are unable to undergo dementia screening [12].

The Oxford Cognitive Screen (OCS) [13] addresses cognitive functioning in a comprehensive manner across several domains to identify impairments beyond the limitations of assessment from language, spatial-temporal, memory, or executive functions commonly affected by stroke. OCS employs a multiple-choice format to enhance the participation of patients with language impairments and vertical layouts to maximize the evaluation of patients who have neglect symptoms. The tool allows for rapid screening within approximately 15 minutes. Other notable advantages of OCS is applicability at bedside and ease of use for patients with hemiplegia that spares a single upper limb. As for the criteria to determine impairment, the OCS uses 5th-percentile cut-offs. The test’s utility is further enhanced by incorporating a “visual snapshot” (Fig. 1), which facilitates communication of the assessment results. The results in the Korean version (K-OCS) are summarized in a snapshot report that displays scores alongside brief descriptions of each subtask, making it easy to understand the patient’s cognitive performance at a glance. The snapshot report serves not only as a tool to indicate areas of impairment but also to record additional notes. For instance, if a patient experiences difficulties only in reading but performs well on other language tasks, only the relevant segment of the “domain” is shaded and explanatory comments can be provided in the adjacent area (e.g., “Deficit in reading only - score 11/15”). Observation of the behavior and medical condition, such as “T-tube limitations or not wearing reading glasses” or “Excited emotional status,” can also be described to provide additional information.

Fig. 1.

Cognitive profile. OCS, Oxford Cognitive Screen.

The initial development of OCS in the United Kingdom was followed by normative studies in various countries, including Italy, Hong Kong, Denmark, Russia, Spain, Portugal, Netherlands, Australia, and Germany [14-22]. Recently, a normative study for K-OCS that considered and adapted cultural and linguistic characteristics has been published (https://process.innovation.ox.ac.uk/clinical/p/ocs/questionnaire/1). The K-OCS was translated and reverse-translated according to a systematic translation procedure in a previous study, adapted to align with the linguistic and cultural characteristics of Korea, and ultimately approved by the original author [23]. As for the criteria to determine impairment, the OCS uses 5th-percentile cut-offs. Detailed information on how to translate and score can be found in the K-OCS normative study [23]. To serve as a useful cognitive assessment tool for stroke, its reliability in patients with stroke must be established. In addition, securing a validity consistent with the results of existing evaluations would be essential. The original OCS has become recognized as a screening test for evaluating PSCIs because of its strong reliability and validity [13], and the latter has been demonstrated in comparison analyses with existing screening tools [10,11]. Additionally, reports have revealed the neurological validity of OCS by demonstrating its ability to detect cognitive impairments characteristic of hemispheric lesions [10].

This study aimed to estimate the feasibility of using K-OCS, which may be a new avenue for post-stroke cognitive screening. For this purpose, we conducted a prospective clinical study and analyzed inter-rater reliability, test–retest reliability, convergent and discriminant validities, specificity, sensitivity, and predictive values, which adopted results of MMSE, MoCA, and other functional evaluation tests.

METHODS

RESULTS

Collectively, 72 participants were recruited (46 male and 26 female). The ages ranged from 31 to 93 years, with a mean age of 62.35±13.80 years and their education level ranged from 1 to 19 years, with a mean of 13.03±3.60 years. The demographic variables and characteristics related to brain damage in the stroke cohort, including the time elapsed after stroke, are presented in Table 1. The onset of the period was less than 3 months in 22 participants (30.6%), between 3 and 12 months in 16 participants (22.2%), and more than 12 months in 34 participants (47.2%). To evaluate inter-rater reliability, 24 participants were randomly selected and their performances in the K-OCS test were video-recorded. Of these, 17 patients were re-tested for evaluation of intra-rater reliability.

Table 1.

Patient sample characteristics (n=72)

Reliability verification

Three raters participated in the inter-rater reliability analysis using a sample of 24 participants. Reliability is classified as follows: a coefficient value of 0–0.4 indicates poor, 0.4–0.6 fair, 0.6–0.75 good, and 0.75–1 excellent reliability [31]. The results suggested excellent inter-rater reliability for the scoring of most subtests in K-OCS (Table 2). Test–retest reliability was analyzed in a sample of 17 participants with an average interval of 2.88±0.82 days between evaluations. We obtained ICC values between 0.913 and 0.994 (p<0.001), which also demonstrated excellent level of agreement (Table 3).

Table 2.

Inter-rater reliability (rater n=3, stroke patient n=24)

Table 3.

Test–retest reliability for each subtest of the K-OCS (n=17)

Validity verification

Data from 72 participants were included in the analysis to verify convergent and discriminant validity correlations. K-OCS and existing tools that assess similar functions demonstrated a moderate-to-strong association (r range, 0.378–0.979, p<0.001; Table 4). We obtained relatively high correlations for K-OCS subscores—imitation (r=0.979), orientation (r=0.849), calculation (r=0.698), and executive mixed tasks (r=0.671). In contrast, relatively low correlations were noted for the executive total (r=|0.386|), recall MoCA (r=0.378), recall MMSE (r=0.430), recognition MoCA (r=0.455), recognition MMSE (r=0.461), episodic memory MoCA (r=0.498), and number writing (r=0.455) tasks (all p<0.01). In the executive total task, a higher total score indicates poorer performance; thus, the correlation is presented as a negative value. To facilitate comparison with findings from other subtests, the values were presented as absolute values, which were then used to compute the r-values; this strategy helps avoiding misinterpretation of negative values that resulted from the reverse scoring system. The results show that correlation is superior for subtasks with content and methods similar to those of existing assessments, while subtasks that highlight the unique features of K-OCS are associated with decreased correlations. As indicated in Supplementary Table S2, the analysis of correlations between cognitive subtests within K-OCS revealed predominantly strong associations, excluding the orientation (r=0.387, p<0.01) and the orientation multiple-choice items (r=0.304, p<0.01). To confirm rationality in the correlation analysis for validity, we chose the FIM motor subscale score as a negative control for discriminant validity. The results mostly revealed no correlation and only a few correlations were noted among the 15 subtasks of K-OCS (Supplementary Table S3).

Table 4.

Convergent validity: correlations between performance on subtests of the K-OCS and other measures of cognition (n=72)

Diagnostic accuracy

In this study, we assessed diagnostic accuracy by comparing the results between the K-OCS cognitive subtests and a representative selective test—K-MoCA memory subtest—which has been widely used in PSCI screening. Specificity, sensitivity, PPV, and NPV were computed and the corresponding values are presented in Table 5. Specificity in the subtasks ranged from 94.8% to 100%, indicating low false positive probability in K-OCS. Sensitivity was relatively lower for specific memory tasks such as recognition (32.2%), episodic memory (25.6%), and calculations in multiple choice questions (35.2%). The PPV for K-OCS, representing the probability of cognitive impairment based on positive results, was notably high for episodic memory and calculation tasks (100%), whereas executive mixed (68.5%) and picture naming tasks (58.2%) showed relatively lower values (Table 5). The NPV of K-OCS, with negative values indicating that cognitive impairment is likely not present, was significantly high for orientation MCQ (98.4%), picture naming (97.1%), and executive mixed tasks (90.2%). In contrast, recognition (65.6%) and episodic memory tasks (69.5%) had comparatively lower values.

Table 5.

Diagnostic accuracy (n=72)

DISCUSSION

The World Health Organization’s Wilson-Jungner criteria, published in 2018, recommend cognitive and emotional screening assessments for all patients with stroke. Additionally, the global focus is on cognitive impairment and rehabilitative interventions after stroke, emphasizing the need to assess function across different cognitive domains [32]. In this study, we reported data on the reliability and validity of K-OCS, which seems appropriate for screening for cognitive impairment in stroke. Verification of the reliability and validity of an assessment tool is crucial for ensuring accuracy and dependability of the measurements in research and clinical practice.

The inter-rater ICC values of K-OCS were 0.914–0.998 across the subset tests, indicating excellent inter-rater reliability [33]. For comparison with prior research results, the Hong Kong-OCS study yielded values of 0.952–1 [15] and the Spanish-OCS study, of 0.790–1 [18]. The test–retest reliability values were 0.906–0.992, which also indicates excellent inter-rater reliability. The original OCS test–retest reliability values were 0.331–0.776 [13], the Hong Kong-OCS study reported values of 0.578–0.989 [15], and the Russian-OCS study, of 0.48–0.96 [17]. There is a notable difference between the present and previous results that showed a significant increase in the re-test score for language [13,15,17]. The average interval between test and retest in this study was 2.88 days, while it was 3.3 days in the original OCS study [13] and sampling was performed at an average of 6 days post-stroke. In contrast, in this study, 30.6% of sampling occurred within 3 months post-stroke. The different condition has likely contributed to the narrower variations in the test–retest scores, particularly in the language domain, in the present study. This is consistent with previous findings, which indicate that patients with acute stroke rapidly recover their language expression abilities early after stroke [34].

Regarding validity, the range of the correlation coefficients between K-OCS subtasks and the corresponding test results of other assessments was |0.386–0.979|, which represents an acceptable level of convergent validity in reference to the values in the original OCS (|0.346–0.902|) [13], Hong Kong-OCS (|0.368–0.732|) [15], Russian-OCS (|0.330–0.950|) [17], and Spanish-OCS (|0.466–0.908|) [18]. The convergent correlations within K-OCS for each subtask value were 0.304–0.676. Similarly, the original OCS study reported values of 0.260–0.680 [13]. Overall, convergent validity appeared somewhat lower than that in other gold-standard neurocognitive assessments. However, according to prior research, MMSE may underestimate non-verbal domains such as memory when evaluating language expression deficits following stroke, and MoCA has also been reported to have limitations in distinguishing between cognitive decline in Alzheimer’s disease and PSCI [35]. The relatively low convergent validity for some K-OCS subtasks may be attributed to the unique ability of this test to measure neuropsychological functions that were indistinguishable with existing cognitive-screening tools. To minimize the impact of aphasia, OCS includes a multiple-choice response format, and to reduce the influence of neglect symptoms, it presents stimuli in vertical layouts. This innovative approach was designed to address the limitations of existing assessment tools by reducing previously recognized confounding effects and enhancing the accuracy of cognitive screening [32,36]. Therefore, compared with existing tools, OCS seems to have enabled a more comprehensive understanding of the intended target component to be measured by enriching insights about the evaluation process.

In this study, discriminant validity assessed the correlation between the FIM-motor task and subtasks in K-OCS; among 15 tasks, we found a moderate degree of correlation in only four (sentence reading, orientation, orientation MCQ, and hearts cancellation). In the original OCS study, subtasks from the Barthel and OCS were used, and correlations were reported only for Broken Hearts [13].

In terms of diagnostic accuracy, relatively low sensitivity scores were observed in certain subtasks, such as recognition (32.2%), episodic memory (25.6%), and the multiple-choice question on calculations (35.2%). These findings may be due to the K-OCS design that differs from traditional assessment tools in aspects that include use of multiple-choice responses and of cued rather than simple recall.

Regarding the study’s limitations, we note that post-injury duration for most of the patients (47.2%) exceeded one year, with 30.6% within three months, and 22.2% between three and 12 months (Table 1). Given that K-OCS is internationally recommended as a screening tool for patients with acute stroke, future research should prioritize validating its use in a large-scale population with acute stroke within the first three months of injury. Another consideration is the absence of a parallel test for K-OCS. Patients with acute stroke often exhibit rapid changes in cognitive function over a short period. Therefore, developing a parallel test with the same content but different items is necessary to enable short-term follow-up. Furthermore, development of parallel forms of testing would be essential to avoid learning effects of repetitive evaluation in patients with rapid recovery.

In conclusion, according to the results of this study, K-OCS overall demonstrated a fair level of validity, supporting its recommendation for clinical use in Korea and conforming to its designation as a stroke-specialized tool with distinct characteristics from those in existing tools. As a tool that is friendly toward common post-stroke symptoms such as aphasia and neglect, K-OCS enables a rapid and comprehensive assessment of patients’ cognitive strengths and weaknesses. Therefore, K-OCS can be applied to patients with stroke in clinical settings, and we expect that future research will continue to validate its clinical utility.

SUPPLEMENTARY MATERIALS

Article information Continued