Natural language processing for automated quantification of bone metastases reported in free-text bone scintigraphy reports

Authors

  • Olivier Q. Groot Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA; Department of Orthopaedic Surgery, University Medical Center Utrecht – Utrecht University, Utrecht, The Netherlands
  • Michiel E. R. Bongers Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA
  • Aditya V. Karhade Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA
  • Neal D. Kapoor Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA
  • Brian P. Fenn Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA
  • Jason Kim Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA
  • J. J. Verlaan Department of Orthopaedic Surgery, University Medical Center Utrecht – Utrecht University, Utrecht, The Netherlands
  • Joseph H. Schwab Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital – Harvard Medical School, Boston, MA, USA

DOI:

https://doi.org/10.1080/0284186X.2020.1819563

Abstract

Background

The widespread use of electronic patient-generated health data has led to unprecedented opportunities for automated extraction of clinical features from free-text medical notes. However, processing this rich resource of data for clinical and research purposes, depends on labor-intensive and potentially error-prone manual review. The aim of this study was to develop a natural language processing (NLP) algorithm for binary classification (single metastasis versus two or more metastases) in bone scintigraphy reports of patients undergoing surgery for bone metastases.

Material and methods

Bone scintigraphy reports of patients undergoing surgery for bone metastases were labeled each by three independent reviewers using a binary classification (single metastasis versus two or more metastases) to establish a ground truth. A stratified 80:20 split was used to develop and test an extreme-gradient boosting supervised machine learning NLP algorithm.

Results

A total of 704 free-text bone scintigraphy reports from 704 patients were included in this study and 617 (88%) had multiple bone metastases. In the independent test set (n = 141) not used for model development, the NLP algorithm achieved an 0.97 AUC-ROC (95% confidence interval [CI], 0.92–0.99) for classification of multiple bone metastases and an 0.99 AUC-PRC (95% CI, 0.99–0.99). At a threshold of 0.90, NLP algorithm correctly identified multiple bone metastases in 117 of the 124 who had multiple bone metastases in the testing cohort (sensitivity 0.94) and yielded 3 false positives (specificity 0.82). At the same threshold, the NLP algorithm had a positive predictive value of 0.97 and F1-score of 0.96.

Conclusions

NLP has the potential to automate clinical data extraction from free text radiology notes in orthopedics, thereby optimizing the speed, accuracy, and consistency of clinical chart review. Pending external validation, the NLP algorithm developed in this study may be implemented as a means to aid researchers in tackling large amounts of data.

Downloads

Download data is not yet available.

Downloads

Additional Files

Published

2020-09-12

How to Cite

Groot, O. Q., Bongers, M. E. R., Karhade, A. V., Kapoor, N. D., Fenn, B. P., Kim, J., … Schwab, J. H. (2020). Natural language processing for automated quantification of bone metastases reported in free-text bone scintigraphy reports. Acta Oncologica, 59(12), 1455–1460. https://doi.org/10.1080/0284186X.2020.1819563

Issue

Vol. 59 No. 12 (2020): Acta Oncologica

Section

Articles