最近人們的消費型態已經從實體店面漸漸轉向網路購物，因此如何讓消費者在眾多商品中快速地搜尋到想買的商品以節省瀏覽的時間是很重要的。特別是在單一類別的商品搜尋中，單靠關鍵字的搜尋往往不能有效地找到符合消費者想要的細節的商品，關鍵字通常只描述商品的屬性像是商品的材質或是商品的類別，如果想要商品的某處有造型或是有裝飾物品就無法用關鍵字去找尋到，像是菱格紋的鞋面或是蝴蝶結造型的裝飾，若是能在一般的關鍵字搜尋之外再加上來自消費者對商品外型需求的簡易描述，像是簡易的素描影像，透過結合這兩種消費者提供的資訊來推薦更符合消費者理想中的商品，以減少消費者瀏覽網頁商品的時間。
在這篇論文中，我們結合簡易素描圖像跟關鍵字去做商品的搜尋和相似商品的推薦，我們透過條件式生成對抗網路將素描圖像跟關鍵字視覺化成彩色影像，再利用此網路鑑別器的其中一層作為生成影像的特徵，最後利用抽取到的特徵來找尋相似的商品影像做推薦。

Recently, consumer’s shopping space transfers from physical store to online shopping websites. How to let consumers search the products they really want in a short time is important. Especially in single category like, shoes, jacket …, only using the texts can’t search the ideal products efficiently. Texts only describe the attributes of the product like, materials or class, it is hard to search the special shape product or some products which contain adornments on the surface through using texts only. If we can fuse another simple information from consumers like sketch image to replenish the lack of texts to promote the products which more fit the consumers’ requirements, it would reduce the searching time greatly.
In this thesis, we combine texts which describe the attributes of the product and sketch which describes the details of the architecture. We use conditional generative adversarial network to generate the color image that fits the attributes and sketch and use one of the discriminator layers as the feature representation to perform the image retrieval.

摘 要 i
Abstract ii
Content iii
Chapter 1 Introduction 1
1.1 Research Background and Motivation 1
1.2 Research Objective 2
1.3 Thesis Organization 3
Chapter 2 Related Work 4
2.1 Attribute Recognition in Image Retrieval 4
2.2 Attribute Recognition in Neural Network 4
2.3 Sketch Image Recognition 5
2.4 Generative Adversarial Network and Conditional Generative Adversarial Network 7
Chapter 3 Proposed method 8
3.1 Network Architecture 8
3.1.1 Generator with Skip 9
3.1.2 Markovian Discriminator (PatchGAN) 9
3.2 Disentangled Representation 10
3.3 Loss Functions 11
3.4 Retrieval Process 12
Chapter 4 Experiments 14
4.1 Dataset 14
4.2 Preprocess Image 15
4.3 Training Process 15
4.4 Testing Results 16
Chapter 5 Conclusion 45
References 46
Appendix 50

[1] W. Di, C. Wah, A. Bhardwaj, R. Piramuthu and N. Sundaresan, “Style Finder: Fine-Grained Clothing Style Recognition and Retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 8-13, 2013.
[2] A. Oliva and A. Torralba, “Modeling the shape of the scene: a holistic representation of the spatial envelope”, International Journal of Conflict and Violence (IJCV), 42(3):145–175, 2001.
[3] J. Wang, Y. Cheng and R. S. Feris, “Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2295 – 2304, 2016.
[4] Z. Liu, P. Luo, X. Wang and X. Tang, “Deep Learning Face Attributes in the Wild”, IEEE International Conf. on Computer Vision (ICCV), pp. 3730-3738, 2015.
[5] H. Zhang, S. Liu, C. Zhang, W. Ren, R. Wang and X. Cao, “SketchNet: Sketch Classification with Web Images”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1105-1113, 2016.
[6] Q. Yu, F. Liu, Y.-Z. Song, T. Xiang, T. M. Hospedales and C. C. Loy, “Sketch Me That Shoe”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 799-807, 2016.
[7] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, “Generative Adversarial Nets”, Neural Information Processing Systems (NIPS), 2014.
[8] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets”, arXiv: 1411.1784, 2014.
[9] P. Isola, J.-Y. Zhu, T. Zhou and A. A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5967 - 5976 , 2017.
[10] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim and J. Choo, “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[11] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 105 – 114, 2017.
[12] S. IIZUKA, E. SIMO-SERRA and H. ISHIKAWA, “Globally and Locally Consistent Image Completion”, ACM Trans. on Graphics (Proceedings of SIGGRAPH), pp. 107:1--107:14, 2017.
[13] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele and H. Lee, “Generative Adversarial Text to Image Synthesis”, International Conf. on Machine Learning (ICML), 2016.
[14] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell and A. A. Efros, “Context Encoders: Feature Learning by Inpainting”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2536 – 2544, 2016.
[15] C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang and H. Li, “High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 4076 – 4084, 2017.
[16] A. Radford, L. Metz and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks”, arXiv: 1511.06434, 2015.
[17] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift”, arXiv: 1502.03167, 2015.
[18] O. Ronneberger, P. Fischer and T. Brox, “U-net: Convolutional networks for biomedical image segmentation”, in The Medical Image Computing and Computer Assisted Intervention Society (MICCAI), pages 234–241. Springer, 2015.
[19] L. Tran, X. Yin and X. Liu, “Disentangled Representation Learning GAN for Pose-Invariant Face Recognition”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1283 – 1292, 2017.
[20] A. Odena, “Semi-Supervised Learning with Generative Adversarial Networks”, arXiv: 1606.01583, 2016.
[21] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford and X. Chen, “Improved Techniques for Training GANs”, Advances in Neural Information Processing Systems (NIPS), 2016.
[22] A. Yu and K. Grauman, “Fine-Grained Visual Comparisons with Local Learning”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 192 – 199, 2014.
[23] A. Yu and K. Grauman, “Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images”, IEEE International Conf. on Computer Vision (ICCV), pp. 5571 – 5580, 2017.
[24] J. Canny, “A Computational Approach to Edge Detection”, IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), pp. 679 – 698, 1986.
[25] D. Kingma and J. Ba, “Adam: A method for stochastic optimization”, International Conf. on Learning Representations (ICLR), 2015.
[26] Q. Yu, Y. Yang, Y.-Z. Song, T. Xiang, and T. Hospedales, “Sketch-a-net that beats humans”, in The British Machine Vision Conference (BMVC), 2015.
[27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and F.-F. Li, “ ImageNet: A large-scale hierarchical image database”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 248 – 255, 2009.
[28] N. Dalal and B. Triggs, “Histograms of Oriented Gradients for Human Detection”, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 886 – 893, 2005.
[29] A. Vedaldi and B. Fulkerson, VLFeat: An Open and Portable Library of Computer Vision Algorithms, http://www.vlfeat.org/, 2008