Agreement Between Single Raters and Team Rating When Applying the International Classification of Functioning, Disability and Health’s Rehabilitation Set

Authors

  • Malan Zhang Department of Exercise Rehabilitation, College of Exercise and Health, Guangzhou Sport University, Guangzhou, China
  • Yun Zhang Department of Rehabilitation, The Fifth Hospital of Xiamen, Xiamen, China
  • Minghong Sui Department of Rehabilitation, Shenzhen Nanshan People’s Hospital, Shenzhen, China
  • Liyin Wang Department of Rehabilitation, Clifford Hospital, Guangzhou, China
  • Ziling Lin Department of Rehabilitation, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China
  • Wei Shen Department of Rehabilitation, GuangDong 999Brain Hospital, Guangzhou, China
  • Jiani Yu Department of Rehabilitation, GuangDong Province Hospital of Chinese Medicine, Guangzhou, China
  • Tiebin Yan Department of Rehabilitation Medicine, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou , China; Guangdong Engineering Technology Research Center for Rehabilitation and Elderly Care

DOI:

https://doi.org/10.2340/jrm.v55.14737

Keywords:

assessment, International Classification of Functioning Disability and Health, team evaluation, rehabilitation

Abstract

Objective: To quantify the agreement between functional assessments by a single rater and a team using the Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set in a clinical situation.

Design: Inter-rater, multi-centre agreement study.

Subjects: A total of 193 adult inpatients admitted to 5 rehabilitation centres at 5 hospitals in China

Methods: The Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set was used by either a single rater or a team to assess 193 patients at 5 Chinese hospitals. Percentage of agreement and quadratic-weighted kappa coefficients were computed. Evaluation times were compared with paired t-tests.

Results: The mean team and individual evaluation times were not significantly different. The percentage of agreement ranged from 46.1% to 94.2% depending on the item, and the quadratic-weighted kappas ranged from 0.43 to 0.92. Eight categories (26.6%) showed a weighted kappa exceeding 0.4, 11 others (36.7%) exceeded 0.6, and another 11 (36.7%) produced kappas of more than 0.8.

Conclusion: Either a single rater or a team of raters can produce valid and consistent ratings when using the Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set to assess patients in a rehabilitation department. The team rating approach is suitable for clinical application.

Downloads

Download data is not yet available.

References

World Health Organization (WHO). The International Classification of Functioning, Disability and Health. Geneva: World Health Organization; 2001.

Vreeman DJ, Richoz C. Possibilities and implications of using the ICF and other vocabulary standards in electronic health records. Physiother Res Int 2013; 20: 210-219.

https://doi.org/10.1002/pri.1559 DOI: https://doi.org/10.1002/pri.1559

Gutenbrunner C, Negrini S, Kiekens C, Zampolini M, Nugraha B. The Global Disability Action Plan 2014-2021 of the World Health Organisation (WHO): a major step towards better health for all people with disabilities. Euro Phys and Rehab Med 2015; 51: 1-4.

Ustun B, Chatterji S, Kostanjsek N. Comments from WHO for the Journal of Rehabilitation Medicine special supplement on ICF Core Sets. J Rehabil Med 2004; 44 Suppl: 7-8.

https://doi.org/10.1080/16501960410015344 DOI: https://doi.org/10.1080/16501960410015344

Cieza A, Ewert T, Ustun TB, Stucki G. Development of ICF core sets for patients with chronic conditions. J Rehabil Med 2004; 44 Suppl: 9-11.

https://doi.org/10.1080/16501960410015353 DOI: https://doi.org/10.1080/16501960410015353

Kraus de Camargo O. International Classification of Functioning, Disability and Health core sets: moving forward. Dev Med Child Neurol 2018; 60: 857-858.

https://doi.org/10.1111/dmcn.13956 DOI: https://doi.org/10.1111/dmcn.13956

Tofani M, Mustari M, Tiozzo E, Dall'Oglio I, Morelli D, Gawronski O, et al. The development of the International Classification of Functioning, Disability and Health for child and youth (ICF-CY) core sets: a systematic review. Disabil Rehabil 2022 Oct 22: 1-10 [Online ahead of print].

Karlsson E, Gustafsson J. Validation of the International Classification of Functioning, Disability and Health (ICF) core sets from 2001 to 2019: a scoping review. Disabil Rehabil 2022; 44: 3736-3748.

https://doi.org/10.1080/09638288.2021.1878562 DOI: https://doi.org/10.1080/09638288.2021.1878562

Cieza A, Oberhauser C, Bickenbach J, Chatterji S, Stucki G. Towards a minimal generic set of domains of functioning and health. BMC Public Health 2014; 14: 218.

https://doi.org/10.1186/1471-2458-14-218 DOI: https://doi.org/10.1186/1471-2458-14-218

Prodinger B, Cieza A, Oberhauser C, Bickenbach J, Üstün TB, Chatterji S, et al. Toward the International Classification of Functioning, Disability and Health (ICF) rehabilitation set: a minimal generic set of domains for rehabilitation as a health strategy. Arch Phys Med Rehab 2016; 97: 875-884.

https://doi.org/10.1016/j.apmr.2015.12.030 DOI: https://doi.org/10.1016/j.apmr.2015.12.030

Li J, Prodinger B, Reinhardt JD, Stucki G. Towards the system-wide implementation of the International Classification of Functioning, Disability and Health in routine practice: lessons from a pilot study in China. J Rehabil Med 2016; 48: 502-507.

https://doi.org/10.2340/16501977-2067 DOI: https://doi.org/10.2340/16501977-2067

Reinhardt JD, Zhang X, Prodinger B, Ehrmann-Bostan C, Selb M, Stucki G, Li J. Towards the system-wide implementation of the International Classification of Functioning, Disability, and Health in routine clinical practice: Empirical findings of a pilot study from mainland China. J Rehabil Med 2016; 48: 515-521.

https://doi.org/10.2340/16501977-2102 DOI: https://doi.org/10.2340/16501977-2102

Ehrmann C, Prodinger B, Stucki G, Cai W, Zhang X, Liu S, et al. ICF generic set as new standard for the system wide assessment of functioning in China: a multicentre prospective study on metric properties and responsiveness applying item response theory. BMJ Open 2018; 8: e021696.

https://doi.org/10.1136/bmjopen-2018-021696 DOI: https://doi.org/10.1136/bmjopen-2018-021696

Senju Y, Mukaino M, Prodinger B, Stucki G. Development of a clinical tool for rating the body function categories of the ICF generic-30/rehabilitation set in Japanese rehabilitation practice and examination of its interrater reliability. BMC Med Res Methodol 2021; 21: 121.

https://doi.org/10.1186/s12874-021-01302-0 DOI: https://doi.org/10.1186/s12874-021-01302-0

Prodinger B, Reinhardt JD, Selb M, Stucki G, Yan T, Zhang X, et al. Towards system-wide implementation of the International Classification of Functioning, Disability and Health (ICF) in routine practice: developing simple, intuitive descriptions of ICF categories in the ICF generic and rehabilitation set. J Rehabil Med 2016; 48: 508-514.

https://doi.org/10.2340/16501977-2066 DOI: https://doi.org/10.2340/16501977-2066

Gao Y, Yan T, You L, Li K. Developing operational items for the International Classification of Functioning, Disability and Health rehabilitation set: the experience from China. Int J Rehabil Res 2018; 41: 20-27.

https://doi.org/10.1097/MRR.0000000000000254 DOI: https://doi.org/10.1097/MRR.0000000000000254

Gao Y, Yan T, You L, Li K, Zhang L, Zhang, M. Psychometric properties of the International Classification of Functioning, Disability and Health Rehabilitation Set: a Rasch analysis. Int J Rehabil Res 2021; 44: 144-151.

https://doi.org/10.1097/MRR.0000000000000463 DOI: https://doi.org/10.1097/MRR.0000000000000463

Zhang M, Yu J, Shen W, Zhang Y, Xiang Y, Zhang X, et al. A mobile app implementing the International Classification of Functioning, Disability and Health Rehabilitation Set. BMC Med Inform Decis Mak 2020; 20: 12-22.

https://doi.org/10.1186/s12911-020-1019-1 DOI: https://doi.org/10.1186/s12911-020-1019-1

Zhang M, Zhang Y, Xiang Y, Lin Z, Shen W, Wang Y, et al. A team approach to applying the International Classification of Functioning, Disability and Health Rehabilitation set in clinical evaluation. J Rehabil Med 2021; 53: jrm00147.

https://doi.org/10.2340/16501977-2756 DOI: https://doi.org/10.2340/16501977-2756

De Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine: a practical guide. Cambridge, UK: Cambridge University Press; 2011.

https://doi.org/10.1017/CBO9780511996214 DOI: https://doi.org/10.1017/CBO9780511996214

Hadkinson H. Evaluation of a mental test score for assessment of mental impairment in the elderly. Age Ageing 1972; 2: 275-279.

https://doi.org/10.1093/ageing/1.4.233 DOI: https://doi.org/10.1093/ageing/1.4.233

Tanglakmankhong K, Hampstead BM, Ploutz-Snyder RJ, Potempa K. Does the Abbreviated Mental Test accurately predict cognitive impairment in Thai older adults? A retrospective study. Pac Rim Int J Nurs Res Thail 2021; 25: 23-33.

World Health Organization (WHO). How to use the ICF: a practical manual for using the International Classification of Functioning, Disability and Health (ICF). Exposure draft for comment. Geneva: World Health Organization; 2013.

Luo WY, Ni P, Chen L, Pan Q, Zhang H, Zhang Y. Development of the ICF-CY set for cardiac rehabilitation after pediatric congenital heart surgery. Frontiers in Pediatrics 2022; 10: 790431.

https://doi.org/10.3389/fped.2022.790431 DOI: https://doi.org/10.3389/fped.2022.790431

Vanbelle S. A new interpretation of the weighted kappa coefficients. Psychometrika 2016; 81: 399-410.

https://doi.org/10.1007/s11336-014-9439-4 DOI: https://doi.org/10.1007/s11336-014-9439-4

Johansson C, Åström S, Kauffeldt A, Carlström E. Daily life dialogue assessment in psychiatric care: Face validity and inter-rater reliability of a tool based on the International Classification of Functioning, Disability and Health. Arch Psychiatr Nurs 2013; 27: 306-311.

https://doi.org/10.1016/j.apnu.2013.08.005 DOI: https://doi.org/10.1016/j.apnu.2013.08.005

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174.

https://doi.org/10.2307/2529310 DOI: https://doi.org/10.2307/2529310

Leonardi M, Lee H, Kostanjsek N, Fornari A, Raggi A, Martinuzzi A, et al. 20 years of ICF International Classification of Functioning, Disability and Health: uses and applications around the world. Int J Environ Res Public Health 2022; 19: 11321.

https://doi.org/10.3390/ijerph191811321 DOI: https://doi.org/10.3390/ijerph191811321

Madden RH, Bundy A. The ICF has made a difference to functioning and disability measurement and statistics. Disabil Rehabil 2019; 41: 1450-1462.

https://doi.org/10.1080/09638288.2018.1431812 DOI: https://doi.org/10.1080/09638288.2018.1431812

Mukaino M, Prodinger B, Yamada S, Senju Y, Izumi SI, Sonoda S, et al. Supporting the clinical use of the ICF in Japan: development of the Japanese version of the simple, intuitive descriptions for the ICF generic-30 set, its operationalization through a rating reference guide, and interrater reliability study. BMC Health Serv Res 2020; 20: 66.

https://doi.org/10.1186/s12913-020-4911-6 DOI: https://doi.org/10.1186/s12913-020-4911-6

Liu S, Reinhardt JD, Zhang X, Ehrmann C, Cai W, Prodinger B. System-wide clinical assessment of functioning based on the International Classification of Functioning, Disability and Health in China: interrater reliability, convergent, known group, and predictive validity of the ICF Generic-6. Arch Phys Med Rehab 2019; 100: 1450-1457.

https://doi.org/10.1016/j.apmr.2018.11.014 DOI: https://doi.org/10.1016/j.apmr.2018.11.014

De Vrieze T, Frippiat J, Deltombe T, Gebruers N, Tjalm WAA, Nevelsteen I, et al. Cross-cultural validation of the French version of the Lymphedema Functioning, Disability and Health Questionnaire for Upper Limb Lymphedema (Lymph-ICF-UL). Disabil Rehab 2020; 28: 1-8.

https://doi.org/10.1037/t83587-000 DOI: https://doi.org/10.1037/t83587-000

Li K, Yan T, You L, Xie S, Li Y, Tang J, et al. The inter-rater reliability of the International Classification of Functioning, Disability and Health set for spinal cord injury nursing. Int J Rehab Res 2016; 39: 240-248.

https://doi.org/10.1097/MRR.0000000000000176 DOI: https://doi.org/10.1097/MRR.0000000000000176

Alvsåker K, Walther SM, Kleffelgård I, Mongs M, Drægebø RA, Keller A. Inter-rater reliability of the Early Functional Abilities scale. J Rehab Med 2011; 43: 892-899.

https://doi.org/10.2340/16501977-0855 DOI: https://doi.org/10.2340/16501977-0855

Dodds TA, Martin DP, Stolov WC, Deyo RA. A validation of the Functional Independence Measurement and its performance among rehabilitation inpatients. Arch Phys Med Rehab 1993; 74: 531-536.

https://doi.org/10.1016/0003-9993(93)90119-U DOI: https://doi.org/10.1016/0003-9993(93)90119-U

Grey N, Kennedy P. The Functional Independence Measure: a comparative study of clinician and self-ratings. Paraplegia 1993; 31: 457-461.

https://doi.org/10.1038/sc.1993.74 DOI: https://doi.org/10.1038/sc.1993.74

Young Y, Fan MY, Hebel JR, Boult C. Concurrent validity of administering the functional independence measure (FIM) instrument by interview. Am J Phys Med Rehab 2009; 88: 766-770.

https://doi.org/10.1097/PHM.0b013e3181a9f1d6 DOI: https://doi.org/10.1097/PHM.0b013e3181a9f1d6

Catz A, Itzkovich M, Steinberg F, Philo O, Ring H, Ronen J, Spasser R, Gepstein R, Tamir A. The Catz-Itzkovich SCIM: a revised version of the Spinal Cord Independence measure. Disability Rehab 2001; 23: 263-268.

https://doi.org/10.1080/096382801750110919 DOI: https://doi.org/10.1080/096382801750110919

Catz A, Itzkovich M, Steinberg F, Philo O, Ring H, Ronen J, et al. Disability assessment by a single rater or a team: a comparative study with the Catz-Itzkovich Spinal Cord Independence measure. J Rehab Med 2002; 34: 226-230.

https://doi.org/10.1080/165019702760279224 DOI: https://doi.org/10.1080/165019702760279224

Leonardi M, Fheodoroff K. Goal Setting with ICF (International Classification of Functioning, Disability and Health) and Multidisciplinary Team Approach in Stroke Rehabilitation. In: Platz T,editor. Clinical Pathways in Stroke Rehabilitation: Evidence-based Clinical Practice Recommendations. Cham (CH): Springer; p.35-56.

https://doi.org/10.1007/978-3-030-58505-1_3 DOI: https://doi.org/10.1007/978-3-030-58505-1_3

Downloads

Published

2023-12-04

How to Cite

Zhang, M., Zhang, Y., Sui, M., Wang, L., Lin, Z., Shen, W., Yu, J., & Yan, T. (2023). Agreement Between Single Raters and Team Rating When Applying the International Classification of Functioning, Disability and Health’s Rehabilitation Set. Journal of Rehabilitation Medicine, 55, jrm14737. https://doi.org/10.2340/jrm.v55.14737

Issue

Vol. 55 (2023)

Section

Original Report

Categories

License

Copyright (c) 2023 Malan Zhang, Yun Zhang, Minghong Sui, Liyin Wang, Ziling Lin, Wei Shen, Jiani Yu, Tiebin Yan

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

All digitalized JRM contents is available freely online. The Foundation for Rehabilitation Medicine owns the copyright for all material published until volume 40 (2008), as from volume 41 (2009) authors retain copyright to their work and as from volume 49 (2017) the journal has been published Open Access, under CC-BY-NC licences (unless otherwise specified). The CC-BY-NC licenses allow third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for non-commercial purposes, provided proper attribution to the original work.
From 2024, articles are published under the CC-BY licence. This license permits sharing, adapting, and using the material for any purpose, including commercial use, with the condition of providing full attribution to the original publication.

Crossref
Scopus
Google Scholar
Europe PMC

Funding data