Development and External Validation of the Machine Learning Models to Predict In-Hospital Cardiac Arrest in the Emergency Department: A Cross-Country Approach

Abstract

Research Question: In Emergency Department (ED) presenting patients within the United States, will our 6 previously internally validated machine-learning (ML) models be able to utilize patient¿s triage data, vitals, chief complain, and demographics to successfully identify those who have had an emergency department-based cardiac arrest (EDCA) event?

Background and Significance: Through our initial approach, we were able to identify utility and predictive strength of ML models for patients at risk of emergency department-based cardiac arrest (EDCA) who presented in an ED in Taiwan. Our cross-country study aims to prove the utility, reliability, and predictive strength of the initial ML models in an ED population within the United States. We hope to provide reliability through an external validation of our initial ML models as a clinical tool to predict and respond appropriately to patients at risk of cardiac arrest who present to the emergency department.

Materials and Methods: We utilized the same training cohort models developed from the database of adult patients at a tertiary training hospital in Taiwan between Jan. 1, 2009, to December 31, 2015. We retrospectively collected data from the ED of a tertiary teaching hospital in the United States between August 31, 2019, to December 31, 2020, to be utilized for external validation as the testing cohort of our ML models. In addition, we trained 6 different ML models in the training cohort using patient features such as triage information and clinical symptoms. We then employed K-fold cross validation and evaluated the performance of our models based on the area under the receiver operating characteristic curve (AUC) in the external validation cohort.

Results: 237,349 and 49,792 patients were included in the training and testing cohort respectively; 477 (0.2%) and 166 (0.3%) were identified to have had an EDCA. All the ML models performed with excellent discrimination based on AUC. Of the constructed ML models, light gradient-boosting machine (LGBM) achieved the best performance of AUC (0.897, 95%, 95% CL: 0.876-0.916) through utility of 7-fold cross validation. There were no significant differences between the constructed models.

Conclusion: Through our study we were able to develop and externally validate our constructed ML models for prediction of EDCA in patients presenting to the ED. Our findings suggest that our ML models have the capabilities to be generalized and applicable as a tool to be used in the ED to predict, prevent, and respond to potential EDCA events based on their discriminatory abilities described in the study.