Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks

Xin Yang, Yi Lin, Zhiwei Wang, Xin Li, Kwang Ting Cheng

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at linyi/Multimodal-Medical-Image-Synthesis.

Original languageEnglish (US)
Article number8736809
Pages (from-to)855-865
Number of pages11
JournalIEEE journal of biomedical and health informatics
Issue number3
StatePublished - Mar 2020
Externally publishedYes


  • Bi-modality
  • generative adversarial learning
  • medical image synthesis
  • semi-supervised learning

ASJC Scopus subject areas

  • Biotechnology
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Health Information Management


Dive into the research topics of 'Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks'. Together they form a unique fingerprint.

Cite this