由於科技的演變，現今有著方便的設備錄製多媒體內容，然而專業的影片編輯者以及電影製作者，可透過錄製的素材以及影片後製編輯技術創造出引發觀眾情緒共鳴的成果。其中以動作和音樂最有著影響人情緒的感染力，尤其在劇場表演中，音樂有著更強的影響力，例如樂團表演演奏的音樂以及指揮家激昂的動作。配合著不同情緒的音樂，表演者的動作會誇張化或是隨著節拍變緩。雖然現今可輕易錄製帶有音樂的影片，但以音樂角度編輯影片仍是一件不易的事。而許多工具可各別編輯音樂以及影片中的影像，然而同時編輯音樂以及影片影像的方式卻是稀少。因此我們提供一套全自動化系統，讓使用者藉由改變背景音樂去操控在預先錄製的影片中人物角色的動作。
在劇場環境中，來源影片是由單一固定的攝影機所拍攝，雖然有著研究致力於改變使用者動作，但他們不同於此環境限制，有著多方視角的影片，可建立良好的虛擬三維模型模擬影片中人物的動作。本研究基於劇場環境中，提出藉由逐漸普及的深度影像攝影機，分析影像層面中無法得到的階層遮擋關係，並產生粗略的三維模型補助建立二點五維模型，最後以影像層面中的變形技術改變表演者的動作幅度。接著我們以不同的骨架來操控指揮家的動作，並可合成出有著許多身體部位互相遮擋的複雜動作。

There are convenient devices capturing multimedia content in modern times. Some professional people, such as moviemaker and video editor, produce attractive and novel multimedia content to invoke emotions of audience with these recorded materials and works about video editing. There is influential power within motion and music.
Specially, music has more influential power in the performance at the theater. The examples are the music performed by the orchestra and passionate motion of the conductor. Motion of the character is exaggerated or restrained with music about different rhythm and emotions.While acquiring the videos with wonderful music is now easy, editing them in terms of visual and musical contents is not trivial. While tremendous efforts have been made in editing visual or musical content along, few works address the scenario of manipulating both contents simultaneously. In this work, we present a fully automatic system that aims at manipulating the character motion in a pre-captured video by simply changing its background music.
The source video is captured by a static camera at the theater. Although there are several pioneers that devoted themselves to manipulate motions of characters, the restriction is different from the environment at the theater. There are multiple viewpoints videos for generating visual 3D model and they can manipulate motion with the 3D model in previous works. According to the environment at the theater, we propose the method between full 3D reconstruction and 2D deformation with the depth and video capture device. We analyze occlusion relation, and create 2.5D model with approximate 3D model. Lastly, we synthesize the manipulated motion of the character by applying image-based deformation. Then, we show videos that there are motions of conductors according to manipulated skeletons, even if there are highly dynamic motions with occlusion conditions.

致謝辭 . . . . . . . . . . . . . . . . . . i
中文摘要 . . . . . . . . . . . . . . . . . ii
Abstract . . . . . . . .. . . . . . . . . iii
List of Tables . . . . . . . . . . . . . vi
List of Figures . . . . . . . . . . . . . vii
1 Introduction . . .. . . . . . . . . . . 1
2 Related Work . . .. . . . . . . . . . . 5
2.1 Video Editing . . . . . . . . . . . . 5
2.2 Motion Re-targeting . . . . . . . . . 7
2.3 2.5D Model . . . . . . . . . . . . . 7
2.4 Deformation . . . . . . . . . . . . . 9
2.5 Texture Synthesis . . . . . . . . . . 10
2.6 Segmentation . . . . .. . . . . . . . 11
3 System Overview . . . . . . . . . . . . 12
3.1 Our Approach . . . . . .. . . . . . . 12
3.2 Framework . . . . . . . . . . . . . . 14
4 Algorithm . . . . . . . . . . . . . . . 16
4.1 Preprocessing . . . . . . . . . . . . 16
4.2 Modeling Phase . . . . . . . . . . . 17
4.2.1 Key Frames Extraction . . . . . . . 17
4.2.2 3D Point Cloud Integration . . .. . 18
4.2.3 Occlusion Analysis . . . . . . . . 20
4.2.4 Image-based 2.5D Modeling . . . . . 22
4.3 Synthesis Phase . . . . . . . . . . . 29
4.3.1 Key Frame Matching . .. . . . . . . 29
4.3.2 Skeleton-driven Image Deformation . 30
5 Results and Discussion . . .. . . . . . 33
6 Conclusion and Future Work . . . . .. . 39
Reference . . . . . . . . . . . . . . . . 40

[1] I. Baran and J. Popovic. Automatic rigging and animation of 3d characters. In ACM Transactions on Graphics (TOG), volume 26, page 72. ACM, 2007.
[2] C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics-TOG, 28(3):24, 2009.
[3] O. Barnich and M. Van Droogenbroeck. Vibe: A universal background subtraction algorithm for video sequences. Image Processing, IEEE Transactions on, 20(6):1709-1724, 2011.
[4] Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 26(9):1124-1137, 2004.
[5] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 23(11):1222-1239, 2001.
[6] J. Chen, S. Paris, J. Wang, W. Matusik, M. Cohen, and F. Durand. The video mesh: A data structure for image-based three-dimensional video editing. In Computational Photography (ICCP), 2011 IEEE International Conference on, pages 1-8. IEEE, 2011.
[7] A. Hornung, E. Dekkers, and L. Kobbelt. Character animation from 2d pictures and 3d motion data. ACM Transactions on Graphics (TOG), 26(1):1, 2007.
[8] T. Igarashi, T. Moscovich, and J. F. Hughes. As-rigid-as-possible shape manipulation. In ACM Transactions on Graphics (TOG), volume 24, pages 1134-1141. ACM, 2005.
[9] A. Jacobson, I. Baran, J. Popovic, and O. Sorkine. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph., 30(4):78, 2011.
[10] V. Kolmogorov and R. Zabin. What energy functions can be minimized via graph cuts? Pattern Analysis and Machine Intelligence, IEEE Transactions on, 26(2):147-159, 2004.
[11] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: image and video synthesis using graph cuts. In ACM Transactions on Graphics (ToG), volume 22, pages 277-286. ACM, 2003.
[12] P. Perez, M. Gangnet, and A. Blake. Poisson image editing. In ACM Transactions on Graphics (TOG), volume 22, pages 313{318. ACM, 2003.
[13] A. Rivers, T. Igarashi, and F. Durand. 2.5 d cartoon models. ACM Transactions on Graphics (TOG), 29(4):59, 2010.
[14] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. In ACM Transactions on Graphics (TOG), volume 23, pages 309-314. ACM, 2004.
[15] S. Schaefer, T. McPhail, and J. Warren. Image deformation using moving least squares. In ACM Transactions on Graphics (TOG), volume 25, pages 533-540. ACM, 2006.
[16] N. Tang, C. Hsu, M. Weng, T. Lin, and H. Liao. Example-based human motion extrapolation and motion repairing using contour manifold. 2014.
[17] V. V. Vazirani. Approximation algorithms. springer, 2001.
[18] F. Xu, Y. Liu, C. Stoll, J. Tompkin, G. Bharaj, Q. Dai, H.-P. Seidel, J. Kautz, and C. Theobalt. Video-based characters: creating new human performances from a multi-view video database. In ACM Transactions on Graphics (TOG), volume 30, page 32. ACM, 2011.
[19] C.-K. Yeh, P. Song, P.-Y. Lin, C.-W. Fu, C.-H. Lin, and T.-Y. Lee. Double-sided 2.5 d graphics. Visualization and Computer Graphics, IEEE Transactions on, 19(2):225-235, 2013.