Validation of Treatment Discontinuation in Claims Data Using NLP and Electronic Health Records

Abstract

Measuring medication discontinuation in claims data primarily relies on the gaps between prescription fills, but such definitions are rarely validated. This study aimed to establish a natural language processing (NLP)-based framework to validate claims-based discontinuation algorithms for commonly used medications against NLP-based reference standards from electronic health records (EHRs). We identified 35,010 patients receiving antipsychotic medications (APMs), benzodiazepines, warfarin, or direct oral anticoagulants (DOACs) from EHRs at the Mass General Brigham in 2007-2020. These EHR data were linked with 82,455 Medicare Part D claims. An NLP-aided chart review was applied to determine medication discontinuation from EHRs (reference standard). In claims data, we defined discontinuation based on a prescription gap larger than 15-90 days (claims-based algorithms). Sensitivity, specificity, and predictive values of claims-based algorithms against the reference standard were calculated. We found that the sensitivity and specificity of 90-day-gap-based algorithms were 0.49 and 0.82 for haloperidol, 0.47 and 0.86 for atypical APMs, 0.54 and 0.75 for benzodiazepines, 0.40 and 0.81 for warfarin, and 0.46 and 0.89 for DOACs, respectively. The corresponding estimates for 15-day-gap-based algorithms were 0.71 and 0.58 for haloperidol, 0.61 and 0.64 for atypical APMs, 0.75 and 0.45 for benzodiazepines, 0.64 and 0.50 for warfarin, and 0.61 and 0.66 for DOACs, respectively. Positive predictive values were primarily affected by medication discontinuation rates and less by gap lengths. In summary, the accuracy of claims-based discontinuation algorithms differs by medications and prescription gaps. This study demonstrates the scalability and utility of the NLP-based validation framework for multiple medications.

Contribution highlights

• This study aimed to establish a natural language processing (NLP)-based framework to validate claims-based discontinuation algorithms for commonly used medications against NLP-based reference standards from electronic health records (EHRs).
• This study demonstrates the scalability and utility of the NLP-based validation framework for multiple medications.

Extracted from the public abstract; no claims have been added.

Code and data

Publication metadata

DOI

10.1093/geroni/igaf122.356

PMCID

PMC12759550

Volume

9

Issue

Supplement_2

Pages / article

igaf122.356

Version relationship

No separate preprint-to-publication relationship is currently recorded.

BibTeX

@article{Yang2025Validationtreatment,
  title = {Validation of Treatment Discontinuation in Claims Data Using NLP and Electronic Health Records},
  author = {Chun-Ting Yang and Kerry Ngan and Dae Hyun Kim and Jie Yang and Jun Liu and Kueiyu Joshua Lin},
  journal = {Innovation in Aging},
  year = {2025},
  doi = {10.1093/geroni/igaf122.356},
  volume = {9},
  number = {Supplement_2},
  pages = {igaf122.356}
}