ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

Stefan Winzeck, Arsany Hakim, Richard McKinley, José A.A.D.S.R. Pinto, Victor Alves, Carlos Silva, Maxim Pisov, Egor Krivov, Mikhail Belyaev, Miguel Monteiro, Arlindo Oliveira, Youngwon Choi, Myunghee Cho Paik, Yongchan Kwon, Hanbyul Lee, Beom Joon Kim, Joong Ho Won, Mobarakol Islam, Hongliang Ren, David RobbenPaul Suetens, Enhao Gong, Yilin Niu, Junshen Xu, John M. Pauly, Christian Lucas, Mattias P. Heinrich, Luis C. Rivera, Laura S. Castillo, Laura A. Daza, Andrew L. Beers, Pablo Arbelaezs, Oskar Maier, Ken Chang, James M. Brown, Jayashree Kalpathy-Cramer, Greg Zaharchuk, Roland Wiest, Mauricio Reyes

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

Original languageEnglish (US)
Article number679
JournalFrontiers in Neurology
Volume9
Issue numberSEP
DOIs
StatePublished - Sep 13 2018
Externally publishedYes

Fingerprint

Benchmarking
Stroke
Research Personnel
Online Systems
Learning
Datasets

Keywords

  • Benchmarking
  • Datasets
  • Deep learning
  • Machine learning
  • MRI
  • Prediction models
  • Stroke
  • Stroke outcome

ASJC Scopus subject areas

  • Neurology
  • Clinical Neurology

Cite this

Winzeck, S., Hakim, A., McKinley, R., Pinto, J. A. A. D. S. R., Alves, V., Silva, C., ... Reyes, M. (2018). ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Frontiers in Neurology, 9(SEP), [679]. https://doi.org/10.3389/fneur.2018.00679

ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. / Winzeck, Stefan; Hakim, Arsany; McKinley, Richard; Pinto, José A.A.D.S.R.; Alves, Victor; Silva, Carlos; Pisov, Maxim; Krivov, Egor; Belyaev, Mikhail; Monteiro, Miguel; Oliveira, Arlindo; Choi, Youngwon; Paik, Myunghee Cho; Kwon, Yongchan; Lee, Hanbyul; Kim, Beom Joon; Won, Joong Ho; Islam, Mobarakol; Ren, Hongliang; Robben, David; Suetens, Paul; Gong, Enhao; Niu, Yilin; Xu, Junshen; Pauly, John M.; Lucas, Christian; Heinrich, Mattias P.; Rivera, Luis C.; Castillo, Laura S.; Daza, Laura A.; Beers, Andrew L.; Arbelaezs, Pablo; Maier, Oskar; Chang, Ken; Brown, James M.; Kalpathy-Cramer, Jayashree; Zaharchuk, Greg; Wiest, Roland; Reyes, Mauricio.

In: Frontiers in Neurology, Vol. 9, No. SEP, 679, 13.09.2018.

Research output: Contribution to journalArticle

Winzeck, S, Hakim, A, McKinley, R, Pinto, JAADSR, Alves, V, Silva, C, Pisov, M, Krivov, E, Belyaev, M, Monteiro, M, Oliveira, A, Choi, Y, Paik, MC, Kwon, Y, Lee, H, Kim, BJ, Won, JH, Islam, M, Ren, H, Robben, D, Suetens, P, Gong, E, Niu, Y, Xu, J, Pauly, JM, Lucas, C, Heinrich, MP, Rivera, LC, Castillo, LS, Daza, LA, Beers, AL, Arbelaezs, P, Maier, O, Chang, K, Brown, JM, Kalpathy-Cramer, J, Zaharchuk, G, Wiest, R & Reyes, M 2018, 'ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI', Frontiers in Neurology, vol. 9, no. SEP, 679. https://doi.org/10.3389/fneur.2018.00679
Winzeck, Stefan ; Hakim, Arsany ; McKinley, Richard ; Pinto, José A.A.D.S.R. ; Alves, Victor ; Silva, Carlos ; Pisov, Maxim ; Krivov, Egor ; Belyaev, Mikhail ; Monteiro, Miguel ; Oliveira, Arlindo ; Choi, Youngwon ; Paik, Myunghee Cho ; Kwon, Yongchan ; Lee, Hanbyul ; Kim, Beom Joon ; Won, Joong Ho ; Islam, Mobarakol ; Ren, Hongliang ; Robben, David ; Suetens, Paul ; Gong, Enhao ; Niu, Yilin ; Xu, Junshen ; Pauly, John M. ; Lucas, Christian ; Heinrich, Mattias P. ; Rivera, Luis C. ; Castillo, Laura S. ; Daza, Laura A. ; Beers, Andrew L. ; Arbelaezs, Pablo ; Maier, Oskar ; Chang, Ken ; Brown, James M. ; Kalpathy-Cramer, Jayashree ; Zaharchuk, Greg ; Wiest, Roland ; Reyes, Mauricio. / ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. In: Frontiers in Neurology. 2018 ; Vol. 9, No. SEP.
@article{c24b4d6d3af749948d9fedb94e73a360,
title = "ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI",
abstract = "Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).",
keywords = "Benchmarking, Datasets, Deep learning, Machine learning, MRI, Prediction models, Stroke, Stroke outcome",
author = "Stefan Winzeck and Arsany Hakim and Richard McKinley and Pinto, {Jos{\'e} A.A.D.S.R.} and Victor Alves and Carlos Silva and Maxim Pisov and Egor Krivov and Mikhail Belyaev and Miguel Monteiro and Arlindo Oliveira and Youngwon Choi and Paik, {Myunghee Cho} and Yongchan Kwon and Hanbyul Lee and Kim, {Beom Joon} and Won, {Joong Ho} and Mobarakol Islam and Hongliang Ren and David Robben and Paul Suetens and Enhao Gong and Yilin Niu and Junshen Xu and Pauly, {John M.} and Christian Lucas and Heinrich, {Mattias P.} and Rivera, {Luis C.} and Castillo, {Laura S.} and Daza, {Laura A.} and Beers, {Andrew L.} and Pablo Arbelaezs and Oskar Maier and Ken Chang and Brown, {James M.} and Jayashree Kalpathy-Cramer and Greg Zaharchuk and Roland Wiest and Mauricio Reyes",
year = "2018",
month = "9",
day = "13",
doi = "10.3389/fneur.2018.00679",
language = "English (US)",
volume = "9",
journal = "Frontiers in Neurology",
issn = "1664-2295",
publisher = "Frontiers Research Foundation",
number = "SEP",

}

TY - JOUR

T1 - ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI

AU - Winzeck, Stefan

AU - Hakim, Arsany

AU - McKinley, Richard

AU - Pinto, José A.A.D.S.R.

AU - Alves, Victor

AU - Silva, Carlos

AU - Pisov, Maxim

AU - Krivov, Egor

AU - Belyaev, Mikhail

AU - Monteiro, Miguel

AU - Oliveira, Arlindo

AU - Choi, Youngwon

AU - Paik, Myunghee Cho

AU - Kwon, Yongchan

AU - Lee, Hanbyul

AU - Kim, Beom Joon

AU - Won, Joong Ho

AU - Islam, Mobarakol

AU - Ren, Hongliang

AU - Robben, David

AU - Suetens, Paul

AU - Gong, Enhao

AU - Niu, Yilin

AU - Xu, Junshen

AU - Pauly, John M.

AU - Lucas, Christian

AU - Heinrich, Mattias P.

AU - Rivera, Luis C.

AU - Castillo, Laura S.

AU - Daza, Laura A.

AU - Beers, Andrew L.

AU - Arbelaezs, Pablo

AU - Maier, Oskar

AU - Chang, Ken

AU - Brown, James M.

AU - Kalpathy-Cramer, Jayashree

AU - Zaharchuk, Greg

AU - Wiest, Roland

AU - Reyes, Mauricio

PY - 2018/9/13

Y1 - 2018/9/13

N2 - Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

AB - Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).

KW - Benchmarking

KW - Datasets

KW - Deep learning

KW - Machine learning

KW - MRI

KW - Prediction models

KW - Stroke

KW - Stroke outcome

UR - http://www.scopus.com/inward/record.url?scp=85055195176&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055195176&partnerID=8YFLogxK

U2 - 10.3389/fneur.2018.00679

DO - 10.3389/fneur.2018.00679

M3 - Article

AN - SCOPUS:85055195176

VL - 9

JO - Frontiers in Neurology

JF - Frontiers in Neurology

SN - 1664-2295

IS - SEP

M1 - 679

ER -