基於新視點內插演算法所做的擬真事後重新對焦需要合成大量的新視點來完成單張重新對焦的照片。為了實踐及時擬真重新對焦，利用此方法所需要內插的新視點數量非常龐大。傳統的新視點內插演算法主要受到高運算量以及不規則記憶體的存取，導致很難用有限的硬體資源來實踐及時擬真重新對焦。
基於一個專門為擬真重新對焦所設計的一維視點合成演算法，我們分析此演算法並提出兩個硬體設計上的挑戰。第一個挑戰是處理不同步的視點輸入信號，這會使得暫存器已及平行運算的設計非常困難。第二個挑戰是如何縮短一條電路上含有迭代迴圈(iteration bound)的關鍵路徑(critical path)。
我們提出利用一個多畫素先進先出的暫存器來解決第一個挑戰。此暫存器分別設計給左右視點使用，來達到獨立更新資料的效果。此外，存多個畫素的目的是為了達到有效率的平行運算。我們也分析了最好的平行度設計是平行度四倍，原因是平行度四倍的設計比起其他平行度能達到較好的硬體資源利用率。對於第二個挑戰，我們利用超前運算(look-ahead)的電路結構已及重排時間(retiming)的電路技巧來大幅縮短電路上的關鍵路徑。利用這兩個方法時鐘頻率可由125M Hz大幅提升至200MHz。
我們實踐了一個支援全高清(Full HD)及時擬真重新對焦的新視點合成引擎。全高清及時擬真重新對焦新視點合成所需要達到的新視點畫素速率是每秒五十億個(5Gpixel/s)。為了達到這個目標，八個一維視點合成器需要同時被使用，也因此消耗許多硬體資源。我們利用一個精準度分析證明可以利用較不精準的除法器在線性內插運算裡，此舉不但合成重新對焦圖片品質幾乎不變，而且能大幅降低所需的硬體資源。基於台積電四零奈米製程所做的電路合成，我們的電路利用六十八點七萬的邏輯閘數來達到每秒三十張的全高清擬真重新對焦的新視點合成引擎。

View-interpolation-based refocusing requires a large number of novel views to generate a single refocused image. To realize real-time refocusing, novel pixels needed to be generated are dramatically high. Conventional view interpolation methods usually suffer from high computation complexity and irregular memory access, and therefore they are difficult to achieve real-time refocusing with limited hardware resources. In this thesis, we aim to design and implement a view synthesis engine for real-time realistic refocusing.
Based on a center-based 1D line scan view synthesis algorithm, which is specifically designed for refocusing, we analyse the algorithm and find two design challenges for hardware implementation. The first challenge is to deal with asynchronous input access of central and right views, which makes the buffer design and the parallelism design difficult. The second challenge is to reduce a long critical path which consists of an iteration bound.
The first challenge is solved by proposed multi-pixel throughput FIFO buffers. The FIFO buffers are applied for central, right and novel views respectively to make data access independent, and the multi-pixel throughput is designed to achieve parallel computation. The parallel designs of the algorithm are analysed and the best parallelism number is four, because it achieves the same output throughput with smaller hardware resources. To reduce the iteration bound, a proposed look-ahead structure and a retiming technique are used to greatly reduce the critical path. As a result, the maximum clock rate can be increased from 125 MHz to over 200 MHz.
A view synthesis engine is implemented to support real-time Full HD refocusing. The required novel pixel rate is over 5 Gpixel/s. To achieve this requirement, eight 1D view synthesis engines are needed. Since so many engines consume enormous hardware resources, a precision analysis shows that we can use the less precise dividers for linear interpolation engines to greatly reduce the number of gate count with only a little distortion. Based on logic synthesis results using TSMC 40nm process, the final design consumes 687k gates, and performs 30 fps for Full HD refocusing.

1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Analysis of 1D View Synthesis Algorithm 11
2.1 Software Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Design Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 Asynchronous Input Data Access . . . . . . . . . . . . . . . 18
2.3.2 Iteration Bound on Critical Path . . . . . . . . . . . . . . . 18 3
3 Architecture Design of 1D View Synthesis Algorithm for High
Novel Pixel Rate 21
3.1 Multi-Pixel Throughput FIFO Design for Synchronization and Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 Analysis of Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 Look-Ahead and Retiming Architecture for Reducing Critical Path 27
3.3.1 Look-Ahead Structure . . . . . . . . . . . . . . . . . . . . . 28
3.3.2 Retiming Technique . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4 Implementation Example for Full HD Refocusing 39
4.1 Novel Pixel Rate Analysis for Full HD Refocusing . . . . . . . . . . 39
4.2 Precision Issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5 Conclusion 51
5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

[1] Ying-Rung Horng, Yu-Cheng Tseng, and Tian-Sheuan Chang, VLSI architecture for real-time HD1080p view synthesis engine," in IEEE Transaction on Circuits and Systems, 2011, vol. 21, pp. 1329 -1340.
[2] Michael Schaffner, Pierre Greisen, Simon Heinzlean, Frank K. Gurkaynak, Hubert Kaeslin, and Aljoscha Smolic, MADmax: A 1080p stereo-to-multiview rendering ASIC in 65 nm CMOS based on image domain warping," in Proc. European Solid-State Circuits Conference, 2013, pp. 61 -64.
[3] M. Harris, Focusing on everything," IEEE Spectrum, vol. 49, no. 5, pp. 44 -50, 2012.
[4] M. Levoy, Light fields and computational imaging," IEEE Computer, vol. 39, no. 8, pp. 46 -55, 2006.
[5] Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz, and Pat Hanrahan, Stanford tech report CTSR 2005-02 light field photography with a hand-held plenoptic camera," 2005.
[6] A. Isaksen, L. McMillan, and S. J. Gortler, Dynamically reparameterized light fields," in SIGGRAPH, 2000, pp. 297 -306.
[7] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy, High performance imaging using large camera arrays," in SIGGRAPH, 2005, pp. 765 -776.
[8] Todor Georgeiv, Ke Colin Zheng, Brian Curless, David Salesin, Shree Nayar, and Chintan Intwala, Spatio-angular resolution tradeoff in integral photography," in Proc. Eurographics Symposium on Rendering, 2006, pp. 263 -272.
[9] P. Rokita, Generating depth of-field effects in virtual reality applications," IEEE Computer Graphics and Applications, vol. 16, no. 2, 1996.
[10] Yuichi Taguchi, Amit Agrawal, Ashok Veeraraghavan, Srikumar Ramalingam, and Ramesh Raskar, Axial-cones: Modeling spherical catadioptric cameras for wide-angle light field rendering," ACM Trans. Graph., vol. 29, no. 6, pp. 172:1 -172:8, Dec. 2010.
[11] Benjamin Huhle, Timo Schairer, Philipp Jenke, and Wolfgang Strasser, Realistic depth blur for images with range data," in Proceedings of the DAGM 2009 Workshop on Dynamic 3D Imaging, 2009, pp. 84 -95.
[12] Chia-Kai Liang, Tai-Hsu Lin, Bing-Yi Wong, Chi Liu, and Homer H. Chen, Programmable aperture photography: Multiplexed light field acquision," in SIGGRAPH, 2008, pp. 55:1 -55:10.
[13] Chao-Tsung Huang, Jui Chin, Hong-Hui Chen, Yu-WenWang, and Liang-Gee Chen, Fast realistic refocusing for sparse light fields," in Proc. International Conference on Acoustics, Speech and Signal Processing, 2015.
[14] Li-Ren Huang, Yu-Wen Wang, and Chao-Tsung Huang, Fast realistic block-based refocusing for sparse light fields," in IEEE International Symposium on Circuits and Systems, 2016.
[15] Christoph Fehn, Depth-image-based rendering, compression and transmission for a new approach on 3D-TV," in Proc. SPIE Conference Stereoscopic Displays Virtual Reality Syst., Dec 2006, vol. 5291, pp. 93 -104.
[16] Ying-Rung Horng, Yu-Cheng Tseng, and Tian-Sheuan Chang, Stereoscopic images generation with directional gaussian filter," in Proc. IEEE International Symposium on Circuits and Systems, May 2010, pp. 2650 -2653.
[17] Chia-Ming Cheng, Shu-Jyuan Li, Shang-Hong Lai, and Jinn-Cherng Yang, Improved novel view synthesis from depth image with large baseline," in Proc. IEEE International Conference on Pattern Recognition, Dec 2008, pp.
1 -4.
[18] Carlos Vzquez, Wa James Tam, and Filippo Speranza, Stereoscopic imaging: filling disoccluded areas in depth image-based rendering," in Proc. SPIE 6392, Oct 2006, vol. 6392.
[19] Chen Wan-yu, Yu-lin Chang, Hsu-kuang Chiu, Shao-yi Chien, and Liang-gee Chen, Real-time depth image based rendering hardware accelerator for advanced three dimensional television system," in Proc. IEEE International Conference on Multimedia and Expo, July 2006, vol. 5291, pp. 2069 -2072.
[20] Manuel Lang, Alexander Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross, Nonlinear disparity mapping for stereoscopic 3d," in ACM SIGGRAPH, July 2010.
[21] George Wolberg, Digital image warping," in IEEE Computer Society press, 1990, vol. 3.
[22] Janusz Konrad and Michael Halle, IEEE signal processing magaziner," in Proc. IEEE International Symposium on Circuits and Systems, Nov 2007, vol. 24, pp. 97 -111.
[23] Test Model 8 of 3D-HEVC and MV-HEVC, ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 document JCT3V-H1003, Apr. 2014.