近年來，神經網路風格轉換因其令人印象深刻的成果而獲得許多關注。然而，它始終難以被良好調控，但在該領域中卻鮮少有人特別著墨於此問題。從先前研究論文的色彩控制方法所產生的神經網路風格轉換成果圖來看，卻還是有著色彩與筆觸不一致及顏色不能正確的被表現的問題。在本篇論文中，我們提出了一個兩步驟流程來解決色彩調控的問題。首先，我們將圖像顏色以加法混色模型所做的初步分類結果，依據 HSV 色彩空間中的色相通道，來進行切分及整併，使圖像顏色得以被分成多個具有代表性的顏色分類。再來，我們所提出的方法以用來解二分圖中最小權重和的匈牙利演算法，去找出顏色分類間的最佳配對組合。接著，我們分別對各個顏色分類配對組合進行色彩轉換。我們在實驗中展示了透過我們提出的方法，可以保留色彩與筆觸間的相依性且同時維持原色特性於神經網路風格轉換輸出圖像。最重要的是，我們也透過實驗來展現神經網路風格轉換之輸入風格圖中顏色面積比例對於神經網路風格轉換的重要性。

Recently, neural style transfer gains a lot of attentions by its impressive results. However, it still suffers from lack of controllability. With previous proposed color controlling methods, the result image from neural style transfer either has inconsistent color and texture rendering, or incorrect rendering colors. In this thesis we propose a two-step workflow to address the color controlling problem. In the first step, we classify image colors into several significant color groups with Hue channel in HSV color space as color entry by merging and splitting the initial color classification based on additive color mixing model. In the second step, our proposed method finds the best match between color classes by using the Hungarian algorithm to solve minimum-weights matching problem in bipartite graph. After that, we conduct color transfer class by class to each color pairs. In our experiments, we demonstrate that our proposed method can effectively generate results of neural style transfer that keep the consistency between color and texture while maintaining color characteristics at the same time.

Acknowledgements
摘要i
Abstract ii
1 Introduction 1
2 Previous Work 3
2.1 Style Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Color Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Problem Definition 13
3.1 Disadvantages From Previous Methods . . . . . . . . . . . . . . . . . . . . . 13
3.2 Summary of Our Observations . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 Formal Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4 Proposed Methods 25
4.1 Why HSV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2 Color Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.3 Color Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4 Color Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5 Implementation 43
6 Experimental Results 47
6.1 Effects of Different Color Spaces . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.2 Effects of Different Cost Functions . . . . . . . . . . . . . . . . . . . . . . . . 50
6.3 Comparison with Previous Methods . . . . . . . . . . . . . . . . . . . . . . . 52
7 Conclusions 61
References 63

[1] L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman, “Controlling
perceptual factors in neural style transfer,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pp. 3985–3993, 2017.
[2] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural
networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 2414–2423, 2016.
[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional
neural networks,” Advances in neural information processing systems, vol. 25,
pp. 1097–1105, 2012.
[4] J. Johnson, A. Alahi, and L. FeiFei,
“Perceptual losses for realtime
style transfer and
superresolution,”
in European conference on computer vision, pp. 694–711, Springer,
2016.
[5] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky, “Texture networks: Feedforward
synthesis of textures and stylized images.,” in ICML, vol. 1, p. 4, 2016.
[6] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” arXiv
preprint arXiv:1610.07629, 2016.
[7] T. Q. Chen and M. Schmidt, “Fast patchbased
style transfer of arbitrary style,” arXiv
preprint arXiv:1612.04337, 2016.
[8] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua, “Stylebank: An explicit representation for
neural image style transfer,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, pp. 1897–1906, 2017.
[9] X. Huang and S. Belongie, “Arbitrary style transfer in realtime
with adaptive instance
normalization,” in Proceedings of the IEEE International Conference on Computer Vision,
pp. 1501–1510, 2017.
[10] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.H.
Yang, “Universal style transfer via
feature transforms,” arXiv preprint arXiv:1705.08086, 2017.
[11] I. Goodfellow, J. PougetAbadie,
M. Mirza, B. Xu, D. WardeFarley,
S. Ozair,
A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in neural information
processing systems, vol. 27, 2014.
[12] J.Y.
Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired imagetoimage
translation using
cycleconsistent
adversarial networks,” in Proceedings of the IEEE international conference
on computer vision, pp. 2223–2232, 2017.
[13] M.Y.
Liu, T. Breuel, and J. Kautz, “Unsupervised imagetoimage
translation networks,”
in Advances in neural information processing systems, pp. 700–708, 2017.
[14] X. Huang, M.Y.
Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised imagetoimage
translation,” in Proceedings of the European conference on computer vision
(ECCV), pp. 172–189, 2018.
[15] H.Y.
Lee, H.Y.
Tseng, J.B.
Huang, M. Singh, and M.H.
Yang, “Diverse imagetoimage
translation via disentangled representations,” in Proceedings of the European conference
on computer vision (ECCV), pp. 35–51, 2018.
[16] T. Karras, S. Laine, and T. Aila, “A stylebased
generator architecture for generative adversarial
networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 4401–4410, 2019.
[17] J. S. De Bonet, “Multiresolution sampling procedure for analysis and synthesis of texture
images,” in Proceedings of the 24th annual conference on Computer graphics and
interactive techniques, pp. 361–368, 1997.
[18] A. A. Efros and T. K. Leung, “Texture synthesis by nonparametric
sampling,” in Proceedings
of the seventh IEEE international conference on computer vision, vol. 2, pp. 1033–
1038, IEEE, 1999.
[19] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, “Image analogies,”
in Proceedings of the 28th annual conference on Computer graphics and interactive techniques,
pp. 327–340, 2001.
[20] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings
of the 28th annual conference on Computer graphics and interactive techniques,
pp. 341–346, 2001.
[21] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, “Graphcut textures: Image and
video synthesis using graph cuts,” Acm transactions on graphics (tog), vol. 22, no. 3,
pp. 277–286, 2003.
[22] Q. Wu and Y. Yu, “Feature matching and deformation for texture synthesis,” ACM Transactions
on Graphics (TOG), vol. 23, no. 3, pp. 364–367, 2004.
[23] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Color transfer between images,”
IEEE Computer graphics and applications, vol. 21, no. 5, pp. 34–41, 2001.
[24] D. L. Ruderman, T. W. Cronin, and C.C.
Chiao, “Statistics of cone responses to natural
images: implications for visual coding,” JOSA A, vol. 15, no. 8, pp. 2036–2045, 1998.
[25] X. Xiao and L. Ma, “Color transfer in correlated color space,” in Proceedings of the 2006
ACM international conference on Virtual reality continuum and its applications, pp. 305–
309, 2006.
[26] F. Pitie, A. C. Kokaram, and R. Dahyot, “Ndimensional
probability density function transfer
and its application to color transfer,” in Tenth IEEE International Conference on Computer
Vision (ICCV’05) Volume 1, vol. 2, pp. 1434–1439, IEEE, 2005.
[27] F. Pitié, A. C. Kokaram, and R. Dahyot, “Automated colour grading using colour distribution
transfer,” Computer Vision and Image Understanding, vol. 107, no. 12,
pp. 123–137,
2007.
[28] O. Frigo, N. Sabater, V. Demoulin, and P. Hellier, “Optimal transportation for exampleguided
color transfer,” in Asian Conference on Computer Vision, pp. 655–670, Springer,
2014.
[29] J.D.
Yoo, M.K.
Park, J.H.
Cho, and K. H. Lee, “Local color transfer between images
using dominant colors,” Journal of Electronic Imaging, vol. 22, no. 3, p. 033003, 2013.
[30] A. Hosni, C. Rhemann, M. Bleyer, C. Rother, and M. Gelautz, “Fast costvolume
filtering
for visual correspondence and beyond,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 35, no. 2, pp. 504–511, 2012.
[31] H. Chang, O. Fried, Y. Liu, S. DiVerdi, and A. Finkelstein, “Palettebased
photo recoloring.,”
ACM Trans. Graph., vol. 34, no. 4, pp. 139–1, 2015.
[32] Q. Zhang, C. Xiao, H. Sun, and F. Tang, “Palettebased
image recoloring using color
decomposition optimization,” IEEE Transactions on Image Processing, vol. 26, no. 4,
pp. 1952–1964, 2017.
[33] P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster
analysis,” Journal of computational and applied mathematics, vol. 20, pp. 53–65, 1987.
[34] J. Tan, J.M.
Lien, and Y. Gingold, “Decomposing images into layers via rgbspace
geometry,”
ACM Transactions on Graphics (TOG), vol. 36, no. 1, pp. 1–14, 2016.
[35] J. Tan, J. Echevarria, and Y. Gingold, “Efficient palettebased
decomposition and recoloring
of images via rgbxyspace
geometry,” ACM Transactions on Graphics (TOG), vol. 37,
no. 6, pp. 1–10, 2018.
[36] M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan, “Deep exemplarbased
colorization,”
ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–16, 2018.
[37] F. Luan, S. Paris, E. Shechtman, and K. Bala, “Deep photo style transfer,” in Proceedings
of the IEEE conference on computer vision and pattern recognition, pp. 4990–4998, 2017.
[38] J. Cho, S. Yun, K. Mu Lee, and J. Young Choi, “Palettenet: Image recolorization with
given color palette,” in Proceedings of the ieee conference on computer vision and pattern
recognition workshops, pp. 62–70, 2017.
[39] C. Xiao, C. Han, Z. Zhang, J. Qin, T.T.
Wong, G. Han, and S. He, “Examplebased
colourization
via dense encoding pyramids,” in Computer Graphics Forum, vol. 39, pp. 20–33,
Wiley Online Library, 2020.
[40] H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics
quarterly, vol. 2, no. 12,
pp. 83–97, 1955.
[41] J. Munkres, “Algorithms for the assignment and transportation problems,” Journal of the
society for industrial and applied mathematics, vol. 5, no. 1, pp. 32–38, 1957.
[42] F. Bourgeois and J.C.
Lassalle, “An extension of the munkres algorithm for the assignment
problem to rectangular matrices,” Communications of the ACM, vol. 14, no. 12,
pp. 802–804, 1971.