Generative adversarial networks (GANs) 訓練容易不穩定而導致模型崩潰 (mode collapse) 。我們在 Boundary Equilibrium Generative Adversarial Network (BEGAN) 上研究 mode collapse 的現象。BEGAN 是目前最先進的 GANs 之 一，可以生成高品質、高多樣性的影像，但是在長時間的訓練之下容易發生 mode collapse 的現象。我們提出 BEGAN with a Constrained Space (BEGAN- CS)，在 loss function 裡面使用 latent-space constraint，使得真假資料的分佈 受限，來降低 mode collapse 的發生。從實驗結果來說，可以顯著的改善訓練 的穩定性、降低 mode collapse 的發生，與此同時，不增加模型的複雜性且不 降低影像的生成品質。我們透過視覺化資料分布的方法來顯示 latent-space constraint 的有效性。在 BEGAN-CS 架構下也有其他額外好處，允許使用小 資料訓練，並得到品質較佳的影像且不易發生 mode collapse。除此之外，也可 以直接生成與查詢影像相似的影像，並且在上面調整影像的屬性。

Generative adversarial networks (GANs) often suffer from unpredictable mode- collapsing during training. We study the issue of mode collapse of Boundary Equilib- rium Generative Adversarial Network (BEGAN), which is one of the state-of-the-art generative models. Despite its potential of generating high-quality images, we find that BEGAN tends to collapse at some modes after a period of training. We pro- pose a new model, called BEGAN with a Constrained Space (BEGAN-CS), which includes a latent-space constraint in the loss function. We show that BEGAN-CS can significantly improve training stability and suppress mode collapse without ei- ther increasing the model complexity or degrading the image quality. Further, we visualize the distribution of latent vectors to elucidate the effect of latent-space con- straint. The experimental results show that our method has additional advantages of being able to train on small datasets and to generate images similar to a given real image yet with variations of designated attributes on-the-fly.

Chinese Abstract i
Abstract ii
Acknowledgments iii
Contents iv
List of Figures v
1 Introduction 1
1.1 Contributions 2
2 Background 4
3 Methods 6
3.1 LatentSpaceAnalysis 7
3.2 ObtainingOptimalz∗inOne-Shot 8
3.3 Disentangled Representation Learning and Application 10
4 Experiments 12
4.1 EffectivenessoftheConstraintLoss 12
4.2 ObservingtheSuddenModeCollapsing 12
4.3 BetterConvergenceonSmallDatasets 14
4.4 ObtainingOptimalz∗inOne-Shot 14
4.5 On-the-FlyRepresentationManipulation 15
5 Conclusion 21

[1] Principal component analysis. Chemometrics and Intelligent Laboratory Sys- tems, 2(1).
[2] Mart ́ın Arjovsky, Soumith Chintala, and L ́eon Bottou. Wasserstein GAN. CoRR, abs/1701.07875, 2017.
[3] David Berthelot, Tom Schumm, and Luke Metz. BEGAN: boundary equilib- rium generative adversarial networks. CoRR, abs/1703.10717, 2017.
[4] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with genera- tive adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 95–104, 2017.
[5] Xi Chen, Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by informa- tion maximizing generative adversarial nets. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2172–2180, 2016.
[6] Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descriptions via a conditional GAN. In IEEE International Con- ference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2989–2998, 2017.
[7] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adver- sarial nets. In Advances in Neural Information Processing Systems 27: Annual
22
Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672–2680, 2014.
[8] JunYoung Gwak, Christopher B. Choy, Animesh Garg, Manmohan Chandraker, and Silvio Savarese. Weakly supervised generative adversarial networks for 3d reconstruction. CoRR, abs/1705.10904, 2017.
[9] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[10] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006.
[11] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progres- sive growing of gans for improved quality, stability, and variation. CoRR, abs/1710.10196, 2017.
[12] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
[13] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1558–1566, 2016.
[14] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunning- ham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 105–114, 2017.
[15] Yijun Li, Sifei Liu, Jimei Yang, and Ming-Hsuan Yang. Generative face comple- tion. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5892–5900, 2017.
23
[16] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
[17] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008.
[18] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spec- tral normalization for generative adversarial networks. CoRR, abs/1802.05957, 2018.
[19] Youssef Mroueh and Tom Sercu. Fisher GAN. In Advances in Neural In- formation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 2510–2520, 2017.
[20] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representa- tion learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2015.
[21] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Rad- ford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural In- formation Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2226–2234, 2016.
[22] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2242–2251, 2017.
[23] Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised se- mantic segmentation using generative adversarial network. In IEEE Interna- tional Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22- 29, 2017, pages 5689–5697, 2017.
[24] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial dis- criminative domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2962–2971, 2017.
[25] Junbo Jake Zhao, Micha ̈el Mathieu, and Yann LeCun. Energy-based generative adversarial network. CoRR, abs/1609.03126, 2016.