Title : Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. Language : English Author, co-author : Bergquist, Timothy [> >] Schaffter, Thomas [> >] Yan, Yao [> >] Yu, Thomas [> >] Prosser, Justin [> >] Gao, Jifan [> >] Chen, Guanhua [> >] Charzewski, Łukasz [> >] Nawalany, Zofia [> >] Brugere, Ivan [> >] Retkute, Renata [> >] Prusokas, Alidivinas [> >] Prusokas, Augustinas [> >] Choi, Yonghwa [> >] Lee, Sanghoon [> >] Choe, Junseok [> >] Lee, Inggeol [> >] Kim, Sunkyu [> >] Kang, Jaewoo [> >] Mooney, Sean D. [> >] Guinney, Justin [> >] Lee, Aaron [> >] Salehzadeh-Yazdi, Ali [> >] Prusokas, Alidivinas [> >] Basu, Anand [> >] Belouali, Anas [> >] Becker, Ann-Kristin [> >] Israel, Ariel [> >] Prusokas, Augustinas [> >] Winter, B. [> >] Moreno, Carlos Vega [> >] Kurz, Christoph [> >] Waltemath, Dagmar [> >] Schweinoch, Darius [> >] Glaab, Enrico [University of Luxembourg > Luxembourg Centre for Systems Biomedicine (LCSB) > Biomedical Data Science] Luo, Gang [> >] Chen, Guanhua [> >] Zacharias, Helena U. [> >] Qiao, Hezhe [> >] Lee, Inggeol [> >] Brugere, Ivan [> >] Kang, Jaewoo [> >] Gao, Jifan [> >] Truthmann, Julia [> >] Choe, Junseok [> >] Stephens, Kari A. [> >] Kaderali, Lars [> >] Varshney, Lav R. [> >] Vollmer, Marcus [> >] Pandi, Maria-Theodora [> >] Gunn, Martin L. [> >] Yetisgen, Meliha [> >] Nath, Neetika [> >] Hammarlund, Noah [> >] Müller-Stricker, Oliver [> >] Togias, Panagiotis [> >] Heagerty, Patrick J. [> >] Muir, Peter [> >] Banda, Peter [> >] Retkute, Renata [> >] Henkel, Ron [> >] Madgi, Sagar [> >] Gupta, Samir [> >] Lee, Sanghoon [> >] Mooney, Sean [> >] Kannattikuni, Shabeeb [> >] Sarhadi, Shamim [> >] Omar, Shikhar [> >] Wang, Shuo [> >] Ghosh, Soumyabrata [] Neumann, Stefan [> >] Simm, Stefan [> >] Madhavan, Subha [> >] Kim, Sunkyu [> >] Von Yu, Thomas [> >] Satagopam, Venkata [] Pejaver, Vikas [> >] Gupta, Yachee [> >] Choi, Yonghwa [> >] Nawalany, Zofia [> >] Charzewski, Łukasz [> >] Lee, Aaron [> >] Salehzadeh-Yazdi, Ali [> >] Prusokas, Alidivinas [> >] Basu, Anand [> >] Belouali, Anas [> >] Becker, Ann-Kristin [> >] Israel, Ariel [> >] Prusokas, Augustinas [> >] Winter, B. [> >] Moreno, Carlos Vega [> >] Kurz, Christoph [> >] Waltemath, Dagmar [> >] Schweinoch, Darius [> >] Luo, Gang [> >] Chen, Guanhua [> >] Zacharias, Helena U. [> >] Qiao, Hezhe [> >] Lee, Inggeol [> >] Brugere, Ivan [> >] Kang, Jaewoo [> >] Gao, Jifan [> >] Truthmann, Julia [> >] Choe, Junseok [> >] Stephens, Kari A. [> >] Kaderali, Lars [> >] Varshney, Lav R. [> >] Vollmer, Marcus [> >] Pandi, Maria-Theodora [> >] Gunn, Martin L. [> >] Yetisgen, Meliha [> >] Nath, Neetika [> >] Hammarlund, Noah [> >] Müller-Stricker, Oliver [> >] Togias, Panagiotis [> >] Heagerty, Patrick J. [> >] Muir, Peter [> >] Banda, Peter [> >] Retkute, Renata [> >] Henkel, Ron [> >] Madgi, Sagar [> >] Gupta, Samir [> >] Lee, Sanghoon [> >] Mooney, Sean [> >] Kannattikuni, Shabeeb [> >] Sarhadi, Shamim [> >] Omar, Shikhar [> >] Wang, Shuo [> >] Neumann, Stefan [> >] Simm, Stefan [> >] Madhavan, Subha [> >] Kim, Sunkyu [> >] Von Yu, Thomas [> >] Pejaver, Vikas [> >] Gupta, Yachee [> >] Choi, Yonghwa [> >] Nawalany, Zofia [> >] Charzewski, Łukasz [> >] Lee, Aaron [> >] Salehzadeh-Yazdi, Ali [> >] Prusokas, Alidivinas [> >] Basu, Anand [> >] Belouali, Anas [> >] Becker, Ann-Kristin [> >] Israel, Ariel [> >] Prusokas, Augustinas [> >] Winter, B. [> >] Moreno, Carlos Vega [> >] Kurz, Christoph [> >] Waltemath, Dagmar [> >] Schweinoch, Darius [> >] Luo, Gang [> >] Chen, Guanhua [> >] Zacharias, Helena U. [> >] Qiao, Hezhe [> >] Lee, Inggeol [> >] Brugere, Ivan [> >] Kang, Jaewoo [> >] Gao, Jifan [> >] Truthmann, Julia [> >] Choe, Junseok [> >] Stephens, Kari A. [> >] Kaderali, Lars [> >] Varshney, Lav R. [> >] Vollmer, Marcus [> >] Pandi, Maria-Theodora [> >] Gunn, Martin L. [> >] Yetisgen, Meliha [> >] Nath, Neetika [> >] Hammarlund, Noah [> >] Müller-Stricker, Oliver [> >] Togias, Panagiotis [> >] Heagerty, Patrick J. [> >] Muir, Peter [> >] Banda, Peter [> >] Retkute, Renata [> >] Henkel, Ron [> >] Madgi, Sagar [> >] Gupta, Samir [> >] Lee, Sanghoon [> >] Mooney, Sean [> >] Kannattikuni, Shabeeb [> >] Sarhadi, Shamim [> >] Omar, Shikhar [> >] Wang, Shuo [> >] Neumann, Stefan [> >] Simm, Stefan [> >] Madhavan, Subha [> >] Kim, Sunkyu [> >] Von Yu, Thomas [> >] Pejaver, Vikas [> >] Gupta, Yachee [> >] Choi, Yonghwa [> >] Nawalany, Zofia [> >] Charzewski, Łukasz [> >] Lee, Aaron [> >] Salehzadeh-Yazdi, Ali [> >] Prusokas, Alidivinas [> >] Basu, Anand [> >] Belouali, Anas [> >] Becker, Ann-Kristin [> >] Israel, Ariel [> >] Prusokas, Augustinas [> >] Winter, B. [> >] Moreno, Carlos Vega [> >] Kurz, Christoph [> >] Waltemath, Dagmar [> >] Schweinoch, Darius [> >] Luo, Gang [> >] Chen, Guanhua [> >] Zacharias, Helena U. [> >] Qiao, Hezhe [> >] Lee, Inggeol [> >] Brugere, Ivan [> >] Kang, Jaewoo [> >] Gao, Jifan [> >] Truthmann, Julia [> >] Choe, Junseok [> >] Stephens, Kari A. [> >] Kaderali, Lars [> >] Varshney, Lav R. [> >] Vollmer, Marcus [> >] Pandi, Maria-Theodora [> >] Gunn, Martin L. [> >] Yetisgen, Meliha [> >] Nath, Neetika [> >] Hammarlund, Noah [> >] Müller-Stricker, Oliver [> >] Togias, Panagiotis [> >] Heagerty, Patrick J. [> >] Muir, Peter [> >] Banda, Peter [> >] Retkute, Renata [> >] Henkel, Ron [> >] Madgi, Sagar [> >] Gupta, Samir [> >] Lee, Sanghoon [> >] Mooney, Sean [> >] Kannattikuni, Shabeeb [> >] Sarhadi, Shamim [> >] Omar, Shikhar [> >] Wang, Shuo [> >] Neumann, Stefan [> >] Simm, Stefan [> >] Madhavan, Subha [> >] Kim, Sunkyu [> >] Von Yu, Thomas [> >] Pejaver, Vikas [> >] Gupta, Yachee [> >] Choi, Yonghwa [> >] Nawalany, Zofia [> >] Charzewski, Łukasz [> >] Lee, Aaron [> >] Salehzadeh-Yazdi, Ali [> >] Prusokas, Alidivinas [> >] Basu, Anand [> >] Belouali, Anas [> >] Becker, Ann-Kristin [> >] Israel, Ariel [> >] Prusokas, Augustinas [> >] Winter, B. [> >] Moreno, Carlos Vega [> >] Kurz, Christoph [> >] Waltemath, Dagmar [> >] Schweinoch, Darius [> >] Luo, Gang [> >] Chen, Guanhua [> >] Zacharias, Helena U. [> >] Qiao, Hezhe [> >] Lee, Inggeol [> >] Brugere, Ivan [> >] Kang, Jaewoo [> >] Gao, Jifan [> >] Truthmann, Julia [> >] Choe, Junseok [> >] Stephens, Kari A. [> >] Kaderali, Lars [> >] Varshney, Lav R. [> >] Vollmer, Marcus [> >] Pandi, Maria-Theodora [> >] Gunn, Martin L. [> >] Yetisgen, Meliha [> >] Nath, Neetika [> >] Hammarlund, Noah [> >] Müller-Stricker, Oliver [> >] Togias, Panagiotis [> >] Heagerty, Patrick J. [> >] Muir, Peter [> >] Banda, Peter [> >] Retkute, Renata [> >] Henkel, Ron [> >] Madgi, Sagar [> >] Gupta, Samir [> >] Lee, Sanghoon [> >] Mooney, Sean [> >] Kannattikuni, Shabeeb [> >] Sarhadi, Shamim [> >] Omar, Shikhar [> >] Wang, Shuo [> >] Neumann, Stefan [> >] Simm, Stefan [> >] Madhavan, Subha [> >] Kim, Sunkyu [> >] Von Yu, Thomas [> >] Pejaver, Vikas [> >] Gupta, Yachee [> >] Choi, Yonghwa [> >] Nawalany, Zofia [> >] Charzewski, Łukasz [> >] Publication date : 2023 Journal title : Journal of the American Medical Informatics Association : JAMIA Peer reviewed : Yes Audience : International ISSN : 1067-5027 e-ISSN : 1527-974X Country : England Keywords : [en] evaluation ; health informatics ; machine learningAbstract : [en] OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.Research centres : Luxembourg Centre for Systems Biomedicine (LCSB): Biomedical Data Science (Glaab Group) Target : Researchers ; Professionals ; Students Permalink : http://hdl.handle.net/10993/55835 DOI : 10.1093/jamia/ocad159 Other URL : https://doi.org/10.1093/jamia/ocad159 Mentions required by the publisher for OA : The original article is available at https://doi.org/10.1093/jamia/ocad159 Commentary : © The Author(s) 2023. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com. 
