Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine.; ; et al in Journal of the American Medical Informatics Association : JAMIA (2023) OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on ... [more ▼] OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI. [less ▲] Detailed reference viewed: 224 (0 UL) Device-based assessment through a mobile application in the Luxembourg Parkinson Cohort; ; Banda, Peter et alin Basal Ganglia (2017, May), 8 Introduction: The project focuses on the integration device-based assessment (DBA) with a mobile application (mPower) into the longitudinal deeply-phenotyped HELP-PD (Health in the Elderly Luxembourgish ... [more ▼] Introduction: The project focuses on the integration device-based assessment (DBA) with a mobile application (mPower) into the longitudinal deeply-phenotyped HELP-PD (Health in the Elderly Luxembourgish Population with a focus on Parkinson’s disease) cohort for patients with Parkinsonism in Luxembourg and the Greater Region to monitor frequency and degree of variation in symptoms of Parkinsonism, to identify potential sources and modulators of variation and to evaluate how symptoms are correlated with these modulators across patients. Methods: We integrate for the first time the mPower iOS app into a deeply phenotyped cohort. mPower is one of the first apps to use Apple’s Research Kit framework and combines a traditional survey-based approach with more granular and precise data gained from a person’s iPhone related to sensor- (e.g. step count, GPS-tracking) or task-based assessments (e.g. finger tapping, tremor detection, sustained phonation, simple gait analysis, memory test). Anonymized longitudinal data is sent to a repository, then retrieved, matched, and correlated with conventional HELP-PD data from a total of 47 screening instruments for motor and non-motor functions in Parkinsonism obtained from annual visits of study participants. 14 patients with clinically confirmed IPD are currently included in the pilot phase. Results/Discussion: We modified the mPower app and successfully integrated it into HELP-PD’s novel database infrastructure, allowing for a wide variety of analyses. The reporting system is able to handle multiple DBAs, with the implementation of an in-depth gait analysis system currently pending. Considerable attention was given to data protection. The system is currently fully functional with the pilot phase having started in June 2016. First correlations with traditional clinical data are planned for early 2017. [less ▲] Detailed reference viewed: 294 (7 UL) |
||