數位影像重對焦可以利用光場資訊，產生出對焦在不同對焦平面、不同光圈大小的影像。這項技術相較於傳統單眼相機提供給使用者更多便利性。擬真影像重對焦影像可以藉由對密集光場直接加權平均獲得，密集光場需要藉由相機陣列來產生，對一般大眾來說無法輕易取得。然而隨著光場相機的推出以及目前多鏡頭手機逐漸盛行，意味者可以更加輕易地取得鬆散光場。只要將鬆散光場內插產生密集光場再進行加權平均也可達到擬真影像重對焦的效果。但倘若內差的光場不足，混疊效果會嚴重影響重對焦影像的品質。目前智慧型手機的前景聚焦是將前後景分離後，對背景進行模糊化模擬對焦於前景的效果，無法產生對背景進行擬真的對焦影像。

本論文提出對經過複數可控金字塔處理後包含五個視點的鬆散光場，進行基於相位的影像重對焦演算法。基於相位的演算法可以將像素在不同頻率的位移紀錄在相位差中，利用相位差可以不用額外深度資訊內插出密集光場。此外，基於相位的計算存在明確的數學公式，對無線密集的光場的加權平均運算可以用積分的形式表示，避免重對焦結果受到混疊效果的影響。透過積分公式的尋找表，可以快速地計算出對焦在不同深度平面的重對焦影像。

基於相位的影像重對焦演算法與目前主流的逐像素的數位重對焦技術比較，在實驗中達到相近的影像品質。另外，基於相位的數位重對焦可以記錄像素在不同頻段的位移差，而逐像素的數位重對焦每個像素只能記錄一個深度資訊，因此，當碰到半透明或反光物件與其他物件重疊的特殊場景，基於相位的方法較能區別不同物件的景深。

Image refocusing can generate images focusing on different focal planes with different aperture ranges by light fields. Comparing to monocular cameras, this technique provides much more convenience for users. Realistically refocused images can be produced directly by the weighted average of dense light fields, but it takes a camera array to gain a dense light field, which is not easily accessible by most of people. With the release of light field cameras and prevailing of multi-lens smartphones, it is increasingly easy to get a sparse light field. Weighted-averaging a dense light field generated by image interpolation on sparse light field is another way to perform realistic image refocusing. However, the aliasing effect will affect the result severely due to insufficient number of interpolated views. Currently, the background blurring effect on smartphones is not realistic image refocusing. It is performed by separating foreground from background and blurring only the background.

In this thesis, we propose a phase-based image refocusing algorithm for 5-view sparse light field processed by complex steerable pyramid. The phase-based algorithm records pixel motions at different frequency levels in phase difference. With phase difference, a dense light field can be generated by image interpolation without additional depth information. Furthermore, there is an explicit mathematical expression in phase-based calculation, so the weighted average calculation on an infinitely dense light field can be expressed by a definite integral formula and address the aliasing effect automatically. Calculation of images focused on different focal planes is quick with a proper lookup table of this formula.

The experimental results of phase-based image refocusing reach similar quality comparing to those of mainstream pixel-based image refocusing. Moreover, phase-based image refocusing records pixel motions at different frequency levels while pixel-based image refocusing maps a single motion to each pixel by disparity map. Therefore, when it comes to special scenes with semi-transparent or semi-reflective objects overlapping the others, phase-based image refocusing is more capable of distinguishing depth of different objects.

1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Complex Steerable Pyramid . . . . . . . . . . . . . . . . . 4
1.2.2 Phase-Based Motion Processing . . . . . . . . . . . . . . . 7
1.2.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . 13
2 View-Synthesis-Based Phase-based Image Refocusing 15
2.1 Basic Ideas of View-Synthesis-Based Image Refocusing . . . . . . 16
2.1.1 Dense Light Field Rendering . . . . . . . . . . . . . . . . 16
2.1.2 Aperture Filter . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Linear Combination on Phase Difference . . . . . . . . . . . . . . 19
2.3 View-synthesis-based Phase-based Image Refocusing . . . . . . . 23
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3 Phase-Integration-Based Image Refocusing 27
3.1 1D Phase-Based Integration Formulas . . . . . . . . . . . . . . . . 28
3.2 Quality of Inteprolation Comparing to Extrapolation . . . . . . . 30
3.3 Image Refocusing by 2D integration on phase . . . . . . . . . . . 32
3.3.1 Square Aperture Image Refocusing by Closed-form Formula 33
3.3.2 Circle Aperture Image Refocusing by Lookup table . . . . 34
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4 Experimental Results 39
4.1 Pyramid Level Test . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2 General Scenes Test . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3 Scenes with Semi-reflective Objects Test . . . . . . . . . . . . . . 45
5 Conclusion and Future Works 51
5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

[1] C. Huang, Y. Wang, L. Huang, J. Chin, and L. Chen, “Fast physically correct refocusing for sparse light fields using block-based multi-rate view interpolation,” IEEE Transactions on Image Processing, vol. 26, no. 2, pp. 603–618, 2017.
[2] S. Meyer, O. Wang, H. Zimmer, M. Grosse, and A. Sorkine-Hornung, “Phase-based frame interpolation for video,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 735–744, 2015.
[3] N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman, “Phase-based video motion processing,” ACM Trans. Graph., vol. 32, no. 4, pp. 80:1–80:10, 2013.
[4] P. Didyk, P. Sitthi-Amorn, W. T. Freeman, F. Durand, and W. Matusik, “Joint view expansion and filtering for automultiscopic 3d displays,” ACM Trans. Graph., vol. 32, no. 6, pp. 221:1–221:8, 2013.
[5] Yue-Yun Li, “Phase-based frame interpolation for non-pixel-wise movements,” 2019.
[6] E. P. Simoncelli and W. T. Freeman, “The steerable pyramid: a flexible architecture for multi-scale derivative computation,” in International Conference on Image Processing, vol. 3, pp. 444–447, 1995.
[7] J. Portilla and E. P. Simoncelli, “A parametric texture model based on joint statistics of complex wavelet coefficients,” International Journal of Computer Vision, vol. 40, no. 1, pp. 49–70, 2000.
[8] W. T. Freeman and E. H. Adelson, “The design and use of steerable filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 9, pp. 891–906, 1991.
[9] S. Meyer, A. Sorkine-Hornung, and M. Gross, “Phase-based modification transfer for video,” in European Conference on Computer Vision, pp. 633–648, 2016.
[10] P. Kellnhofer, P. Didyk, S. Wang, P. Sitthi-Amorn, W. T. Freeman, F. Durand, and W. Matusik, “3dtv at home: Eulerian-lagrangian stereo-to-multiview conversion,” ACM Trans. Graph., vol. 36, no. 4, pp. 146:1–146:13, 2017.
[11] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4d light fields,” in Asian Conference on Computer Vision, pp. 19–34, 2016.
[12] S. Wanner, S. Meister, and B. Goldlücke, “Datasets and benchmarks for densely sampled 4d light fields,” in Vision, Modeling, and Visualization, pp. 225–226, 2013.
[13] C. Huang, J. Chin, H. Chen, Y. Wang, and L. Chen, “Fast realistic refocusing for sparse light fields,” in IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1176–1180, 2015.
[14] “The (new) stanford light field archive,” http://lightfield.stanford.edu/.
[15] “Middlebury stereo datasets,” http://vision.middlebury.edu/stereo/data/.