TY - JOUR
T1 - Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks
AU - Yang, Xin
AU - Lin, Yi
AU - Wang, Zhiwei
AU - Li, Xin
AU - Cheng, Kwang Ting
N1 - Funding Information:
Manuscript received December 29, 2018; revised May 11, 2019; accepted June 9, 2019. Date of publication June 14, 2019; date of current version March 6, 2020. This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700402, in part by the National Natural Science Foundation of China under Grant 61502188, in part by the Hubei Provincial Natural Science Foundation under Grant ZRMS2017000375, in part by the Wuhan Science and Technology Bureau under Award 2017010201010111, in part by the Fundamental Research Funds for the Central Universities under Grant 2019kfyRCPY118, and in part by the Program for HUST Acadamic Frontier Youth Team. (Corresponding author: Yi Lin.) X. Yang, Y. Lin, and Z. Wang are with the School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China (e-mail:,xinyang2014@hust.edu.cn; lin_yi@hust.edu.cn; zhiweiwang@hust.edu.cn).
Publisher Copyright:
© 2013 IEEE.
PY - 2020/3
Y1 - 2020/3
N2 - In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at https://github.com/hust- linyi/Multimodal-Medical-Image-Synthesis.
AB - In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at https://github.com/hust- linyi/Multimodal-Medical-Image-Synthesis.
KW - Bi-modality
KW - generative adversarial learning
KW - medical image synthesis
KW - semi-supervised learning
UR - http://www.scopus.com/inward/record.url?scp=85081724042&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85081724042&partnerID=8YFLogxK
U2 - 10.1109/JBHI.2019.2922986
DO - 10.1109/JBHI.2019.2922986
M3 - Article
C2 - 31217133
AN - SCOPUS:85081724042
VL - 24
SP - 855
EP - 865
JO - IEEE Journal of Biomedical and Health Informatics
JF - IEEE Journal of Biomedical and Health Informatics
SN - 2168-2194
IS - 3
M1 - 8736809
ER -