光場相機的應用已經逐漸成為計算攝影領域的主流研究，自由視點合成更是其研究的重點之一。自由視點合成可以在光場相機所擷取的視點影像中任意產生欲觀看的視點影像。近年來，在depth-image-based rendering(DIBR)的自由視點合成相關研究中，在光場架構下需要與視點相同數量的disparity map來完成視點合成，如果這些disparity map皆由disparity search演算法或sensor來產生，無論是在成本與計算複雜度上都過於高昂。所以如何使用較少的disparity map來完成自由視點合成且達到良好的品質則顯得相當重要。
本論文探討的是，在光場架構中僅使用一張disparity map完成自由視點合成。我們研究發現在DIBR演算法中進行視點合成時，仍然需要欲合成方向的另一視點disparity map，所以我們將已有的disparity map根據disparity值進行forward warping至欲使用的視點，但是經過forward warping後的圖會有破洞產生，破洞會影響視點合成的品質，必須修補破洞後才能進行後續的視點合成，因此，如何修補破洞並得到良好的品質是非常重要的。此外，破洞修補也為我們演算法的運算瓶頸，需要硬體才能即時運算，為此我們也對此進行硬體架構的探討。以下將破洞修補分為演算法與架構部份做說明:在演算法部分，我們使用加權眾數濾波器(weighted mode filter, WMF)修補破洞，但是常見的WMF會造成前景資訊向外擴散且運算複雜度大，因此，我們提出優化方法，同時提升修補品質與降低運算複雜度；在硬體部份，傳統加權眾數濾波器的硬體架構於SRAM的使用數量與視差標籤(disparity label)範圍成正比，使得面積與成本隨著範圍增加而增加，因此，我們提出cacheable histogram的演算法與硬體架構用於WMF，達到減少SRAM數量與面積的目的。
本論文所提出的演算法，可以在光場架構下僅使用一張disparity map完成2D自由視點合成。此外，我們提出的硬體架構與演算法，使硬體面積皆小於傳統WMF架構；當應用於具有256個視差標籤之高畫素光場時，我們的硬體面積僅是其14 %的面積，在品質方面仍可以得到相近的disparity map與視點合成結果。

The applications of light field camera have gradually become the mainstream research in computational photography, and free viewpoint synthesis has become a hot spot in the research. Free viewpoint synthesis can synthesize any viewpoints which are required from light field camera. In recent years, the related research of free viewpoint synthesis of DIBR (depth-image-based rendering) has revealed that the number of disparity maps is as same as the number of viewpoints in the light field configuration. If these disparity maps are all from disparity search algorithm or sensor, the algorithm complexity and the hardware costs are too high. Therefore, it is important to synthesize good quality of viewpoint by using less disparity maps.
This thesis emphasizes the use of only one disparity map to complete free viewpoint synthesis in light field configuration. We find out that DIBR algorithm still needs another corresponding disparity map, so we forward warp the disparity map we already have to the corresponding viewpoint. However, holes appear after forward warping, and they affect the quality of viewpoint synthesis. Holes must be filled in order to synthesize viewpoint afterwards. Therefore, how to fill holes and achieve good quality are critical. In addition, hole filling algorithm is also the bottleneck in our system, so we design hardware architecture to reach real-time computation. We divide hole filling into two aspects: algorithm and architecture. In algorithm, we use weighted mode filter(WMF) to fill holes, but general WMF makes foreground information spread and requires high complexity. Our proposed method optimizes the quality of hole filling and reduce computational complexity. In hardware, the size of SRAM is proportional to the number of disparity labels in traditional hardware architecture of WMF. Therefore, we propose the algorithm and architecture of cacheable histogram to WMF. Proposed method can reduce the size of SRAM and hardware area.
Our proposed algorithm can complete 2D free viewpoint synthesis by using only one disparity map. We also propose hardware architecture and algorithm to make hardware area smaller than traditional WMF architecture. In the high resolution light field which has 256 disparity labels, our hardware area is only 14% of traditional hardware of WMF, but our method can still reach similar quality of disparity map and synthesized viewpoint.

致謝 I
摘要 III
Abstract V
Content VII
List of Figures IX
List of Tables XI
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Related Work 2
1.3 System Overview 6
Chapter 2 Algorithm Study on Disparity Map Synthesis for Free Viewpoint Synthesis of Sparse Light Field 9
2.1 Free Viewpoint Synthesis of Sparse Light Field 9
2.1.1 1D Viewpoint Synthesis Algorithm in Our System 10
2.1.2 2D Viewpoint Synthesis Algorithm Extend from 1D Viewpoint Synthesis Algorithm 11
2.2 Weighted Mode Filter for Hole Filling 14
2.3 Timing Profiling 19
Chapter 3 Algorithm Optimization for Weighted Mode Filter 21
3.1 Asymmetric Weighted Mode Filter for Hole Filling 21
3.2 Window Subsampling for Complexity Reduction 23
3.3 Objective Quality and Complexity Comparison 24
Chapter 4 Architecture Optimization of Weighted Mode Filter 31
4.1 Discussion on Prior Art 31
4.2 Proposed Cacheable Histogram Architecture 33
4.3 Study on Scan Direction 36
4.4 Implementation Result 38
Chapter 5 Conclusion 43
Reference 45

[1]. M. Tanimoto, M.P. Tehrani, T. Fujii, and T. Yendo, “FreeViewpoint TV,” in IEEE Signal Processing Magazine, pp. 67-76, 2011.
[2]. B. M. Smith, L. Zhang, H. Jin, and A. Agarwala, “Light field video stabilization,” in IEEE ICCV, 2009.
[3]. M. Schaffner, P. Greisen, S. Heinzle, F. K. Gurkaynak, H. Kaeslin, and A. Smolic, “MADmax: A 1080p Stereo-to-Multiview Rendering ASIC in 65 nm CMOS based on Image Domain Warping,” in Proceedings of ESSCIRC, pp. 61–64, 2013.
[4]. N. Dodgson, “Autostereoscopic 3-D displays,” in IEEE Computer, pp. 31–36, 2005.
[5]. P. E. Debevec, C. J. Taylor, and J. Malik, "Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach," in ACM Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 11-20, 1996.
[6]. M. Levoy, “Light fields and computational imaging,” in IEEE Computer, pp. 46-55, 2006
[7]. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The Lumigraph,” in ACM SIGGRAPH 1996 Conference Proceedings, pp. 43–54, 1996.
[8]. C. Kim, H. Zimmer, Y. Pritch, A. Sorkine-Hornung and M. H. Gross, “Scene reconstruction from high spatio-angular resolution light fields,” in ACM Trans. Graph., 2013 .
[9]. C. Buehler, M. Bosse, L. McMillan, S. J. Gortler and M. F. Cohen,”Unstructured Lumigraph rendering,” in ACM SIGGRAPH 2001 Conference Proceedings, pp. 425–432, 2001.
[10]. J. Gomes, L. Darsa, B. Costa and L. Velho,”Warping and Morphing of Graphical Objects,” in Morgan Kaufmann Publishers, 1999.
[11]. S. M. Seitz and C. R. Dyer, “View morphing,” in Proc. SIGGRAPH, pp. 21-30, 1996.
[12]. Y. Mori, N. Fukushima, T. Yendo, T. Fujii and M. Tanimoto, “View generation with 3D warping using depth information for FTV,” in Signal Processing: Image Communication, pp. 65-72, 2009.
[13]. “Kinect v2 sensor,”
[online] http://link.springer.com/article/10.1007/s11042-016-3523-y
[14]. D. Min, J. Lu and M. N. Do, “Depth video enhancement based on weighted mode filtering,” in IEEE Transactions on Image Processing, pp. 1176-1190, 2012.
[15]. L.-D. Chen, Y.-L. Hsiao, and C.-T. Huang, "VLSI Architecture Design of Weighted Mode Filter for Full-HD Depth Map Upsampling at 30fps," in IEEE International Symposium on Circuits and Systems, 2016.
[16]. Test Model 8 of 3D-HEVC and MV-HEVC, ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 document JCT3V-H1003, Apr. 2014.
[17]. “HCI benchmark datasets,”
[online] http://hci.iwr.uni-heidelberg.de/HCI/Research/LightField/