Inter-Rater Reliability of Novice Linkers Using an Innovative Sequential Iterative Linking Method to Link Prosthetic Outcomes to The International Classification of Functioning, Disability and Health
DOI:
https://doi.org/10.2340/jrm.v55.2409Keywords:
reliability, agreement, International Classification of Functioning, Disability and Health, ICF, linking, prosthetics, amputation, outcomesAbstract
Objective: When linking outcomes to the International Classification of Functioning, Disability and Health (ICF), inter-rater reliability is typically assessed at the conclusion of the linking process. This method does not allow for iterative evaluation and adaptations that would improve inter-rater reliability as novices gain experience. This pilot study aims to quantify the inter-rater reliability of novice linkers when using an innovative, sequential, iterative linking method to link prosthetic outcomes to the ICF.
Methods: Across 5 sequential rounds, 2 novices independently linked outcomes to the ICF. A consensus discussion followed each round that informed refinement of the customized ICF linking rules. The inter-rater reliability was calculated for each round using Gwet’s agreement coefficient (AC1).
Results: A total of 1,297 outcomes were linked across 5 rounds. At the end of round 1 inter-rater reliability was high (AC1 = 0.74, 95% confidence interval (95% CI) 0.68–0.80). At the end of round 3, interrater reliability (AC1 = 0.84, 95% CI 0.80–0.88) was significantly improved and marked the point of consistency where further improvements in inter-rater reliability were not statistically significant.
Conclusion: A sequential iterative linking method provides a learning curve that allows novices to achieve high-levels of agreement through consensus discussion and iterative refinement of the customized ICF linking rules.
LAY ABSTRACT
Outcomes are commonly used in healthcare and research to evaluate the effect of an intervention or treatment, such as the effect a prosthesis has on the ability to walk in the community or participate in activities. Cataloguing outcomes using well-established classification systems, such as the International Classification of Functioning, Disability and Health, is important, as it allows outcomes and research to be described using an internationally understood and agreed language. This study aimed to describe an innovative approach to cataloguing outcomes to the ICF, using a method that provides novices with a learning opportunity. In using this innovative method novices were able to catalogue outcomes to the ICF framework with a similar degree of reliability as experts. This will reduce the barriers to novices conducting this type of research in the future.
Downloads
References
Fayed N, Cieza A, Edmond Bickenbach J. Linking health and health-related information to the ICF: a systematic review of the literature from 2001 to 2008. Disabil Rehabil 2011; 33: 1941-1951.
https://doi.org/10.3109/09638288.2011.553704 DOI: https://doi.org/10.3109/09638288.2011.553704
Cieza A, Brockow T, Ewert T, Amman E, Kollerits B, Chatterji S, et al. Linking health-status measurements to the international classification of functioning, disability and health. J Rehabil Med 2002; 34: 205-210.
https://doi.org/10.1080/165019702760279189 DOI: https://doi.org/10.1080/165019702760279189
Cieza A, Geyh S, Chatterji S, Kostanjsek N, Ustün B, Stucki G. ICF linking rules: an update based on lessons learned. J Rehabil Med 2005; 37: 212-218.
https://doi.org/10.1080/16501970510040263 DOI: https://doi.org/10.1080/16501970510040263
Cieza A, Fayed N, Bickenbach J, Prodinger B. Refinements of the ICF Linking Rules to strengthen their potential for establishing comparability of health information. Disabil Rehabil 2019; 41: 574-583.
https://doi.org/10.3109/09638288.2016.1145258 DOI: https://doi.org/10.3109/09638288.2016.1145258
Cieza A, Stucki G. Content comparison of health-related quality of life (HRQOL) instruments based on the International Classification of Functioning, Disability and Health (ICF). Qual Life Res 2005; 14: 1225-1237.
https://doi.org/10.1007/s11136-004-4773-0 DOI: https://doi.org/10.1007/s11136-004-4773-0
Soberg HL, Sandvik L, Ostensjo S. Reliability and applicability of the ICF in coding problems, resources and goals of persons with multiple injuries. Disabil Rehabil 2008; 30: 98-106.
https://doi.org/10.1080/09638280701216862 DOI: https://doi.org/10.1080/09638280701216862
Sigl T, Cieza A, Brockow T, Chatterji S, Kostanjsek N, Stucki G. Content comparison of low back pain-specific measures based on the International Classification of Functioning, Disability and Health (ICF). Clin J Pain 2006; 22: 147-153.
https://doi.org/10.1097/01.ajp.0000155344.22064.f4 DOI: https://doi.org/10.1097/01.ajp.0000155344.22064.f4
Stamm T, Geyh S, Cieza A, Machold K, Kollerits B, Kloppenburg M, et al. Measuring functioning in patients with hand osteoarthritis - content comparison of questionnaires based on the International Classification of Functioning, Disability and Health (ICF). Rheumatology (Oxford) 2006; 45: 1534-1541.
https://doi.org/10.1093/rheumatology/kel133 DOI: https://doi.org/10.1093/rheumatology/kel133
Castro S, Ferreira T, Dababnah S, Pinto AI. Linking autism measures with the ICF-CY: Functionality beyond the borders of diagnosis and interrater agreement issues. Dev Neurorehabil 2013; 16: 321-331.
https://doi.org/10.3109/17518423.2012.733438 DOI: https://doi.org/10.3109/17518423.2012.733438
Chen S, Tao J, Tao Q, Fang Y, Zhou X, Chen H, et al. Rater experience influences reliability and validity of the Brief International Classification of Functioning, Disability, and Health Core Set for Stroke. J Rehabil Med 2016; 48: 265-272.
https://doi.org/10.2340/16501977-2063 DOI: https://doi.org/10.2340/16501977-2063
Starrost K, Geyh S, Trautwein A, Grunow J, Ceballos-Baumann A, Prosiegel M, et al. Interrater reliability of the extended ICF core set for stroke applied by physical therapists. Phys Ther 2008; 88: 841-851.
https://doi.org/10.2522/ptj.20070211 DOI: https://doi.org/10.2522/ptj.20070211
Wu SM, Whiteside U, Neighbors C. Differences in inter-rater reliability and accuracy for a treatment adherence scale. Cogn Behav Ther 2007; 36: 230-239.
https://doi.org/10.1080/16506070701584367 DOI: https://doi.org/10.1080/16506070701584367
Clarke L, Ridgewell E, Dillon M. Measuring the benefits of prosthetic interventions in health economic evaluations. Part 1: identifying and linking outcomes to the ICF framework. Under peer review 2021.
Clarke L, Dillon M, Shiell A. Health economic evaluation in orthotics and prosthetics: a systematic review protocol. Syst Rev 2019; 8: 152.
https://doi.org/10.1186/s13643-019-1066-9 DOI: https://doi.org/10.1186/s13643-019-1066-9
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097.
https://doi.org/10.1371/journal.pmed.1000097 DOI: https://doi.org/10.1371/journal.pmed.1000097
International Organization for Standardization (ISO). Assistive products for persons with disability - Classification and terminology (ISO 9999), ISO/TC 173/SC 2. Geneva, Switzerland: ISO; 2016.
World Health Organization. ICF e-Learning Tool. Geneva: WHO; 2020 [accessed 2020 June 12]. Available from: https://www.icf-elearning.com/
World Health Organization. Towards a common language for Functioning, Disability and Health ICF. Geneva: WHO; 2002.
World Health Organization. How to use the ICF: a practical manual for using the International Classification of Functioning, Disability and Health (ICF). Exposure draft for comment. Geneva: WHO; 2013.
Deathe AB, Wolfe DL, Devlin M, Hebert JS, Miller WC, Pallaveshi L. Selection of outcome measures in lower extremity amputation rehabilitation: ICF activities. Disabil Rehabil 2009; 31: 1455-1473.
https://doi.org/10.1080/09638280802639491 DOI: https://doi.org/10.1080/09638280802639491
Hebert JS, Wolfe DL, Miller WC, Deathe AB, Devlin M, Pallaveshi L. Outcome measures in amputation rehabilitation: ICF body functions. Disabil Rehabil 2009; 31: 1541-1554.
https://doi.org/10.1080/09638280802639467 DOI: https://doi.org/10.1080/09638280802639467
Radhakrishnan S, Kohler F, Gutenbrunner C, Jayaraman A, Li J, Pieber K, et al. The use of the International Classification of Functioning, Disability and Health to classify the factors influencing mobility reported by persons with an amputation: an international study. Prosthet Orthot Int 2017; 41: 412-419.
https://doi.org/10.1177/0309364616652016 DOI: https://doi.org/10.1177/0309364616652016
Radhakrishnan S, Kohler F, Gutenbrunner C, Jayaraman A, Pieber K, Li J, et al. Mobility in persons with lower extremity amputations and influencing factors: using the International Classification of Functioning, Disability and Health to quantify expert views. Prosthet Orthot Int 2019; 43: 88-94.
https://doi.org/10.1177/0309364618792714 DOI: https://doi.org/10.1177/0309364618792714
Theeven PJ, Hemmen B, Brink PR, Smeets RJ, Seelen HA. Measures and procedures utilized to determine the added value of microprocessor-controlled prosthetic knee joints: a systematic review. BMC Musculoskelet Disord 2013; 14: 1-12.
https://doi.org/10.1186/1471-2474-14-333 DOI: https://doi.org/10.1186/1471-2474-14-333
Xu J, Kohler F, Dickson H. Systematic review of concepts measured in individuals with lower limb amputation using the International Classification of Functioning, Disability and Health as a reference. Prosthet Orthot Int 2011; 35: 262-268.
https://doi.org/10.1177/0309364611412821 DOI: https://doi.org/10.1177/0309364611412821
World Health Organization. ICF Browser. Geneva: WHO; 2017. Available from: https://apps.who.int/classifications/icfbrowser/
World Health Organization. ICF: International Classification of Functioning, Disability and Health. Geneva: WHO; 2001.
Gwet KL. Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters. Gaithersburg, MD: Advanced Analytics, LLC; 2014. Available from: https://www.agreestat.com/book4/9780970806284_prelim_chapter1.pdf.
Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990; 43: 551-558.
https://doi.org/10.1016/0895-4356(90)90159-M DOI: https://doi.org/10.1016/0895-4356(90)90159-M
Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990; 43: 543-549.
https://doi.org/10.1016/0895-4356(90)90158-L DOI: https://doi.org/10.1016/0895-4356(90)90158-L
Zec S, Soriani N, Comoretto R, Baldi I. High agreement and high prevalence: the paradox of Cohen's Kappa. Open Nurs J 2017; 11: 211-218.
https://doi.org/10.2174/1874434601711010211 DOI: https://doi.org/10.2174/1874434601711010211
R Foundation. irrCAC: Computing Chance-Corrected Agreement Coefficients (CAC) 2019. Available from: https://CRAN.R-project.org/package=irrCAC
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174.
https://doi.org/10.2307/2529310 DOI: https://doi.org/10.2307/2529310
Ogonowski J, Kronk R, Rice C, Feldman H. Inter-rater reliability in assigning ICF codes to children with disabilities. Disabil Rehabil 2004; 26: 353-361.
https://doi.org/10.1080/09638280410001658658 DOI: https://doi.org/10.1080/09638280410001658658
Kohler F, Connolly C, Sakaria A, Stendara K, Buhagiar M, Mojaddidi M. Can the ICF be used as a rehabilitation outcome measure? A study looking at the inter- and intra-rater reliability of ICF categories derived from an ADL assessment tool. J Rehabil Med 2013; 45: 881-887.
https://doi.org/10.2340/16501977-1194 DOI: https://doi.org/10.2340/16501977-1194
Hagberg K, Brånemark R, Hägg O. Questionnaire for Persons with a Transfemoral Amputation (Q-TFA): initial validity and reliability of a new outcome measure. J Rehabil Res Dev 2004; 41: 695-706.
https://doi.org/10.1682/JRRD.2003.11.0167 DOI: https://doi.org/10.1682/JRRD.2003.11.0167
Coenen M, Stamm TA, Stucki G, Cieza A. Individual interviews and focus groups in patients with rheumatoid arthritis: a comparison of two qualitative methods. Qual Life Res 2012; 21: 359-370.
https://doi.org/10.1007/s11136-011-9943-2 DOI: https://doi.org/10.1007/s11136-011-9943-2
Cerniauskaite M, Quintas R, Boldt C, Raggi A, Cieza A, Bickenbach JE, et al. Systematic literature review on ICF from 2001 to 2009: its use, implementation and operationalisation. Disabil Rehabil 2011; 33: 281-309.
https://doi.org/10.3109/09638288.2010.529235 DOI: https://doi.org/10.3109/09638288.2010.529235
Additional Files
Published
How to Cite
License
Copyright (c) 2023 Leigh Clarke, Emily Ridgewell, Xia Li, Michael P. Dillon
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
All digitalized JRM contents is available freely online. The Foundation for Rehabilitation Medicine owns the copyright for all material published until volume 40 (2008), as from volume 41 (2009) authors retain copyright to their work and as from volume 49 (2017) the journal has been published Open Access, under CC-BY-NC licences (unless otherwise specified). The CC-BY-NC licenses allow third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for non-commercial purposes, provided proper attribution to the original work.
From 2024, articles are published under the CC-BY licence. This license permits sharing, adapting, and using the material for any purpose, including commercial use, with the condition of providing full attribution to the original publication.
