近年來，多鏡頭的應用已經是智慧型手機拍照的新趨勢，拍完照再重新對焦的功能，儼然成為一種取得含有淺景深影像的新方式。但目前市面上未有能同時保有高畫質且擬真的影像與即時生成的技術，所以在此方面的研究與應用的推廣越來越加重要，並為多種應用開闢新的可能性，例如：動態追焦、虛擬實境顯示。因此在本論文中，我們設計一個適用於全高清鬆散光場的重新對焦引擎，此引擎以區塊為單位來合成大量新視點，進而完成重新對焦的影像。

我們將光場資訊儲存在晶片外部的記憶體，使用鬆散光場可以有效減少了相對於緊密光場所需的資料儲存量，但需要內插的新視點數量非常龐大，而內插的新視點所需的大量記憶體成本，我們利用多層級的排程，達到不論內插的新視點數量，也能固定晶面內部的記憶體用量。此外，為了能即時產生重新對焦的影像，我們使用了兩種不同平行度的內插新視點的引擎：四個像素-四條線的單視點產生引擎（Single-view Renderer）、四個像素-四條線-四個視點的多視點產生引擎（Multi-view Renderer），以達到該設計的高吞吐量。

我們基於台積電四零奈米製程實作了一個支援全高清即時擬真重新對焦的超大型積體電路，此電路使用了271KB的晶片內部記憶體以及2.3百萬的邏輯閘，而此晶片的面積為1.9x1.9 mm2。運作在200MHz時，它可以每秒提供5.4G新視點畫素的吞吐量來即時產生出全高清的重新對焦的影像。

In recent years, multi-lenses cameras becomes the mainstream of smart-phones. The ability of changing focal planes after capturing the light field enables users to get the shallow-depth-of-field images. However, the fast realistic refocusing technique with high resolution is still a challenge. Therefore, this research topic becomes more and more important, and it will open more possibilities for many applications, such as auto-panning video and virtual reality display. In the thesis, we propose a Full-HD refocusing engine, which uses block-partitioned sparse light fields to synthesize numerous viewpoints and get a refocused image.

We store the sparse light fields in off-chip memory, which have less storage demand than dense light fields. But the number of rendered viewpoints is tremendous, and it results in high requirement of on-chip memory. The schedule of hierarchical pipeline is designed to synthesize numerous viewpoints using fixed number of on-chip memories. Besides, we apply two types of view-renderers with different parallelism to support the high throughput, including single-view renders in four-pixel/four-line parallelism and multi-view renders in four-pixel/four-line/four-view parallelism.

We implemented a VLSI circuit to support real-time Full-HD refocusing using TSMC 40nm technology process with 271-KB on-chip memory and 2.3M logic gates. The core area is 1.9x1.9 mm2. When synthesized at 200 MHz, it can deliver 5.4G rendered pixels per second and 40M refocused pixels per second to support Full-HD refocusing in real-time.

Abstract ii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Pixel Splatting for Dense Light Fields . . . . . . . . . . . . 6
1.2.2 Block-based View Interpolation for Sparse Light Fields . . . 6
1.2.3 Depth-dependent Image Blurring for Single View . . . . . . 7
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Algorithm Analysis of Block-based Refocusing 13
2.1 Software Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Analysis of Largest Refocusing Block Size . . . . . . . . . . . . . . 22
3 System Architecture of Block-Based Refocusing 27
3.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Hierarchical Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.1 Block-level Pipeline . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.2 Aperture-level Pipeline . . . . . . . . . . . . . . . . . . . . . 28
3.2.3 View-level Pipeline . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4 Implementation of Block-based Refocusing 35
4.1 On-Chip Memory Usage Analysis with Different Block Size . . . . . 35
4.2 Read Throughput Analysis of Off-Chip Memory with Different Loading Method . . 37
4.3 Physical Implementation Result . . . . . . . . . . . . . . . . . . . . . . . 38
5 Conclusion and Future Work 46

[1] Lytro Camera, “https://www.lytro.com/,” .
[2] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy, “High performance imaging using large camera arrays,” in SIGGRAPH, 2005, pp.765–776.
[3] Mathieu Bredif Gene Duval Mark Horowitz Ren Ng, Marc Levoy and Pat Hanrahan, “Light field photography with a hand-held plenoptic camera,” 2005.
[4] Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar, “Picam: An ultra-thin high performance monolithic camera arrays,” in SIGGRAPH Asia, 2013, pp. 166:1–166:13.
[5] M. Harris, “Focusing on everything,” IEEE Spectrum, vol. 49, no. 5, pp.44–50, 2012.
[6] M. Levoy, “Light fields and computational imaging,” IEEE Computer, vol.39, no. 8, pp. 46–55, 2006.
[7] C. Zhou and S. K. Nayar, “Computational cameras: Convergence of optics and processing,” IEEE Trans. Image Processing, vol. 20, no. 12, pp. 3322–3340, 2011.
[8] M. Levoy and Pat Hanrahan, “Light field rendering,” July 1996.
[9] H.-Y. Shum S.C. Chan and K.-T. Ng, “Image-based rendering and synthesis,” IEEE Signal Process. Mag., vol. 24, no. 6, pp. 22–33, 2007.
[10] P. Rokita, “Generating depth of-field effects in virtual reality applications,” IEEE Computer Graphics and Applications, vol. 16, no. 2, 1996.
[11] Joe Demers, “Depth of field: A survey of techniques,” in GPU Gems, pp.375–390. Addison-Wesley, 2004.
[12] Todor Georgeiv, Ke Colin Zheng, Brian Curless, David Salesin, Shree Nayar, and Chintan Intwala, “Spatio-angular resolution tradeoff in integral photography,” in Proc. Eurographics Symposium on Rendering, 2006, pp. 263–272.
[13] Chia-Kai Liang, Tai-Hsu Lin, Bing-Yi Wong, Chi Liu, and Homer H. Chen, “Programmable aperture photography: Multiplexed light field acquisition,” in SIGGRAPH, 2008, pp. 55:1–55:10.
[14] Hong-Hui Chen Yu-Wen Wang Chao-Tsung Huang, Jui Chin and Liang-Gee Chen, “Fast realistic refocusing for sparse light fields,” in Proc. International Conference on Acoustics, Speech and Signal Processing, 2015.
[15] Li-Ren Huang, Yu-Wen Wang, and Chao-Tsung Huang, “Fast realistic blockbased refocusing for sparse light fields,” in IEEE International Symposium on Circuits and Systems, 2016.
[16] Li-Ren Huang-Jui Chin Chao-Tsung Huang, Yu-Wen Wang and Liang-Gee Chen, “Fast physically-correct refocusing for sparse light fields using blockbased multi-rate view interpolation,” in IEEE Transactions on Image Processing, 2017.
[17] Yu-Wen Wang, “A fast realistic block-based refocusing for sparse light fields,” in NTHU, 2015.
[18] B. Curless J. Fiss and R. Szeliski, “Refocusing plenoptic images using depthadaptive splatting,” IEEE International Conference on Computational Photography, pp. 1-9., 2014.
[19] Test Model 8 of 3D-HEVC and MV-HEVC, ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 document JCT3V-H1003, Apr. 2014.
[20] F. Moreno-Noguer, P.N. Belhumeur, and S.K. Nayar, “Active refocusing of images and videos,” ACM Trans. Graph., Aug 2007.
[21] Middleblury stereo dataset, “http://vision.middlebury.edu/stereo/data/,” .
[22] HCI benchmark dataset, “http://hci.iwr.uni-heidelberg.de/hci/,” .
[23] Stanford light field archive, “http://lightfield.stanford.edu/lfs.html,” .
[24] Po-Han Chen, “Vlsi architecture of multi-view/multi-line view synthesis engine for high-performance block-based refocusing,” in NTHU, 2018.
[25] Li-Ren Huang, “Analysis and VLSI architecture design of 1D view synthesis algorithm for fast realistic refocusing,” in NTHU, 2016.
[26] T.-S. Chang J.-C. Tuan and C.-W. Jen, “On the data reuse and memory bandwidth analysis for full-search block-matching VLSI architecture,” IEEETrans. Circuits Syst. Video Technol., vol. 12, no. 1, pp. 61–72, 2002.
[27] Yu-Cheng Tseng Ying-Rung Horng and Tian-Sheuan Chang, “VLSI architecture for real-time hd1080p view synthesis engine,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no. 9, pp. 1329–1340, 2011.
[28] Y. Tseng F. Chang and T. Chang, “A 94fps view synthesis engine for hd1080p video,” Visual Communications and Image Processing (VCIP), pp. 1–4, 2011.
[29] P. Tsung et al., “A 216fps 4096×2160p 3DTV set-top box soc for free-viewpoint 3DTV applications,” IEEE International Solid-State Circuits Conference, pp.124–126, 2011.
[30] Simon Heinzlean Frank K. Gurkaynak-Hubert Kaeslin Michael Schaffner, Pierre Greisen and Aljoscha Smolic, “Madmax: A 1080p stereo-to-multiview rendering asic in 65 nm cmos based on image domain warping,” Proceedings of the ESSCIRC (ESSCIRC), pp. 61–64, 2013.