動態 3D 點雲技術提供了沉浸式使用者體驗，因此在體積視訊串流應用中變得越來越受歡迎。 在透過網路進行串流傳輸時，點雲幀可能會因遺失或延遲的封包而受到大幅度的視覺品質下降。 為了解決這個問題，我們提出了首個錯誤隱蔽的流水線框架，該框架包括五個階段：前處理、匹配、運動估計、預測和後處理。 每個階段都可以開發替代演算法，而不同階段的演算法可以混合搭配成管線，以進行端到端效能評估。 我們討論了設計目標，並為每個階段提出了多種演算法。 然後，我們使用具有不同特徵的動態 3D 點雲序列對這些演算法進行了定量比較。 基於比較結果，我們提出了四個代表性管線，分別適用於：（i）不同程度的運動變化，即輕微與顯著，以及（ii）不同的應用需求，即高品質與低開銷。 對我們提出的管線進行了廣泛的端到端評估，結果表明，它們在以下兩方面的隱蔽品質明顯優於3D 幀複製方法：（i）3D 指標中，GPSNR 提高了多達5.32 dB，CPSNR 提高 了1.7 dB；以及（ii）2D 指標中，PSNR 提高了多達2.22 dB，SSIM 提高了0.06，VMAF 提高了11.67。 此外，一項對 15 名受試者的用戶研究表明，我們性能最佳的管線在消耗僅 8.55% 的運行時間的情況下，實現了對最先進的基於學習的插值演算法 100% 的偏好勝率。

Dynamic 3D point clouds enable an immersive user experience and thus have become increasingly more popular in volumetric video streaming applications. When being streamed over best-effort networks, point cloud frames may suffer from lost or late packets, leading to non-trivial quality degradation. To solve this problem, we proposed the very first error concealment pipeline framework, which comprises five stages: pre-processing, matching, motion estimation, prediction, and post-processing. Alternative algorithms can be developed for each stage, while algorithms of different stages could be mixed and matched into pipelines for end-to-end performance evaluations. We discussed the design goal and proposed multiple algorithms for each stage. These algorithms were then quantitatively compared using dynamic 3D point cloud sequences with diverse characteristics. Based on the comparison results, we proposed four representative pipelines for: (i) diverse degrees of motion variance, i.e., minor versus significant, and (ii) different application requirements, i.e., high quality versus low overhead. Extensive end-to-end evaluations of our proposed pipelines demonstrated their superior concealed quality over the 3D frame-copy method in both: (i) 3D metrics, by up to 5.32 dB in GPSNR and 1.7 dB in CPSNR; as well as (ii) 2D metrics, by up to 2.22 dB in PSNR, 0.06 in SSIM, and 11.67 in VMAF. Adding to that, a user study with 15 subjects indicated that our best-performing pipeline achieved a 100\% preference winning rate over the state-of-the-art learning-based interpolation algorithms while consuming merely up to 8.55\% of running time.

Abstract i
中文摘要ii
1 Introduction 1
1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Background 5
2.1 3D Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Point Cloud Compression . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Error Concealment Problem . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Related Work 14
3.1 Error Concealment of 2D Video Frames . . . . . . . . . . . . . . . . . . 14
3.2 Inpainting of 2D/3D Content . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Completion of 2D/3D Content . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Error Concealment of 3D Point Clouds . . . . . . . . . . . . . . . . . . . 16
4 Error Concealment Pipeline Framework 18
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Design Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3 Matching Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3.1 Alternative Solutions . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3.2 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.4 Motion Estimation Stage . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4.1 Alternative Solutions . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4.2 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.5 Prediction Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.5.1 Alternative Solutions . . . . . . . . . . . . . . . . . . . . . . . . 26
4.5.2 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.6 Pre-Processing and Post-Processing Stages . . . . . . . . . . . . . . . . . 30
5 Experiments 33
5.1 Our Proposed Representative Pipelines . . . . . . . . . . . . . . . . . . . 33
5.2 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.4.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6 Concluding Remarks 66
6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Bibliography 68

[1] A. Aaron, Z. Li, M. Manohara, J. Y. Lin, E. C.-H.Wu, and C.-C. J. Kuo. Challenges
in cloud based ingest and encoding for high quality streaming media. In Proc. of the
IEEE International Conference on Image Processing (ICIP’15), pages 1732–1736,
Quebec City, Canada, September 2015.
[2] A. Akhtar, Z. Li, G. Van der Auwera, and J. Chen. Dynamic point cloud interpolation.
In Proc. of the IEEE International Conference on Acoustics, Speech, and
Signal Processing (ICASSP’22), pages 2574–2578, Singapore, May 2022.
[3] T. Breckon and R. Fisher. Three-dimensional surface relief completion via nonparametric
techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence,
30(12):2249–2255, December 2008.
[4] J. Chen, J. Yi, M. Kahoush, E. Cho, and Y. Cho. Point cloud scene completion
of obstructed building facades with generative adversarial inpainting. Sensors,
20(18):5029:1–5029:27, September 2020.
[5] J. Davis, S. Marschner, M. Garr, and M. Levoy. Filling holes in complex surfaces using
volumetric diffusion. In Proc. of the First International Symposium on 3D Data
Processing Visualization and Transmission (3DPVT’02), pages 428–441, Padova,
Italy, June 2002.
[6] E. d’Eon, B. Harrison, T. Myers, and P. Chou. 8i voxelized full bodies–a voxelized
point cloud dataset, 2017. http://plenodb.jpeg.org/pc/8ilabs/.
[7] E. Dumic and L. da Silva Cruz. Subjective quality assessment of V-PCC compressed
dynamic point clouds degraded by packet losses. Sensors, 23(12):5623:1–5623:28,
June 2023.
[8] Z. Fu and W. Hu. Dynamic point cloud inpainting via spatial-temporal graph learning.
IEEE Transaction on Multimedia, 23:3022–3034, March 2021.
[9] D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai.
An overview of ongoing point cloud compression standardization activities: Videobased
(V-PCC) and geometry-based (G-PCC). APSIPA Transactions on Signal and
Information Processing, 9:e13:1–e13:17, April 2020.
[10] G. Guennebaud, B. Jacob, et al. Eigen v3, 2010. http://eigen.tuxfamily.org.
[11] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. Deep learning for
3D point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 43(12):4338–4364, December 2021.
[12] M. S. Hamid, N. Abd Manap, R. A. Hamzah, and A. F. Kadmin. Stereo matching
algorithm based on deep learning: A survey. Journal of King Saud University-
Computer and Information Sciences, 34(5):1663–1673, May 2022.
[13] G. Haßlinger and O. Hohlfeld. The Gilbert-Elliott model for packet loss in real time
services on the Internet. In Proc. of the GI/ITG Conference-Measurement, Modelling
and Evaluation of Computer and Communication Systems (MMB’08), pages 1–15,
Dortmund, Germany, March 2008.
[14] H. Hoppe. Poisson surface reconstruction and its applications. In Proc. of the ACM
Symposium on Solid and Physical Modeling (SPM’08), pages 10–10, Stony Brook,
NY, June 2008.
[15] W. Hu, Z. Fu, and Z. Guo. Local frequency interpretation and non-local selfsimilarity
on graph for point cloud inpainting. IEEE Transactions on Image Processing,
28(8):4087–4100, March 2019.
[16] I.-C. Huang. Composing error concealment pipelines for dynamic 3D point cloud
streaming, 2023. https://github.com/Huang-I-Chun/error concealment pipeline.
[17] J.-B. Huang, S. B. Kang, N. Ahuja, and J. Kopf. Image completion using planar
structure guidance. ACM Transactions on graphics, 33(4):1–10, July 2014.
[18] T.-K. Hung. On error concealment of dynamic 3D point cloud streaming. Master’s
thesis, National Tsing Hua University, 2022.
[19] T.-K. Hung, I.-C. Huang, S. R. Cox, W. T. Ooi, and C.-H. Hsu. Error concealment
of dynamic 3D point cloud streaming. In Proc. of the ACM Multimedia (MM’22),
pages 3134–3142, Lisboa, Portugal, October 2022.
[20] A. K. Ingale and D. U. J. Real-time 3D reconstruction techniques applied in dynamic
scenes: A systematic literature review. Computer Science Review, 39:100338:1–
100338:13, February 2021.
[21] Methodology for the subjective assessment of the quality of television pictures
ITU-R Recommendation BT.500-13. Document, International Telecommunication
Union, 2012.
[22] E. S. Jang, M. Preda, K. Mammou, A. M. Tourapis, J. Kim, D. B. Graziosi, S. Rhyu,
and M. Budagavi. Video-based point-cloud-compression standard in MPEG: From
evidence collection to committee draft. IEEE Signal Process Mag, 36(3):118–123,
May 2019.
[23] M. Kazemi, M. Ghanbari, and S. Shirmohammadi. A review of temporal video error
concealment techniques and their suitability for HEVC and VVC. Multimedia Tools
and Applications, 80:12685–12730, January 2021.
[24] M. Kazhdan and H. Hoppe. Screened Poisson surface reconstruction. ACM Transactions
on Graphics, 32(3):(29):1–(29):13, June 2013.
[25] D. Kumari and K. Kaur. A survey on stereo matching techniques for 3D vision in
image processing. Int. J. Eng. Manuf, 4:40–49, July 2016.
[26] P. Lancaster and K. Salkauskas. Surfaces generated by moving least squares methods.
Mathematics of computation, 37(155):141–158, July 1981.
[27] S. Lee, S. W. Oh, D. Won, and S. J. Kim. Copy-and-paste networks for deep video
inpainting. In Proc. of the IEEE/CVF International Conference on Computer Vision
(ICCV’19), pages 4413–4421, Seoul, Korea, October 2019.
[28] L. Li, Z. Li, V. Zakharchenko, J. Chen, and H. Li. Advanced 3D motion prediction
for video-based dynamic point cloud compression. IEEE Transactions on Image
Processing, 29:289–302, August 2020.
[29] P. Liepa. Filling holes in meshes. In Proc. of the Eurographics/ACM SIGGRAPH
symposium on Geometry processing (SGP’03), pages 200–205, Goslar, Germany,
June 2003.
[30] F. Lozes, A. Elmoataz, and O. L´ezoray. Partial difference operators on weighted
graphs for image processing on surfaces and point clouds. IEEE Transactions on
Image Processing, 23(9):3896–3909, July 2014.
[31] F. Lozes, A. Elmoataz, and O. L´ezoray. PDE-based graph signal processing for 3-D
color point clouds: Opportunities for cultural heritage. IEEE Signal Process Mag,
32(4):103–111, June 2015.
[32] MPEG. G-PCC codec description v12. Technical Report N00151, ISO/IEC JTC
1/SC 29/WG 7 MPEG Output Document, 2021.
[33] MPEG 3DG. Call for proposals for point cloud compression v2. Technical Report
N16763, ISO/IEC, JTC 1/SC 29/WG 11, 2017.
[34] Common test conditions for V3C and V-PCC. Document, ISO/IEC
JTC1/SC29/WG7 MPEG 3D Graphics Coding, 2020.
[35] V-PCC test model v11. Document, ISO/IEC JTC1/SC29/WG11 MPEG 3DG, 2020.
[36] M. Pauly, N. Mitra, J. Giesen, M. Gross, and L. Guibas. Example-based 3D scan
completion. In Proc. of the Symposium on Geometry Processing (SGP’05), pages
23–32, Vienna, Austria, July 2005.
[37] R. Rusu and S. Cousins. 3D is here: Point Cloud Library (PCL). In Proc. of the
ICRA’11, May 2011.
[38] S. Studio. Factory 42 and hold the world., 2023. https://www.immerseuk.org/
case-study/factory-42-and-hold-the-world.
[39] Y.-C. Sun, I.-C. Huang, Y. Shi, W. T. Ooi, C.-Y. Huang, and C.-H. Hsu. A dynamic
3D point cloud dataset for immersive applications. In Proc. of the ACM Multimedia
Systems Conference (MMsys’23), pages 376–383, Vancouver, Canada, June 2023.
[40] J. van der Hooft, H. Amirpour, M. T. Vega, Y. Sanchez, R. Schatz, T. Schierl, and
C. Timmerer. A tutorial on immersive video delivery: From omnidirectional video to
holography. IEEE Communications Surveys & Tutorials, 25(2):1336–1375, March
2023.
[41] J. Verdera, V. Caselles, M. Bertalmio, and G. Sapiro. Inpainting surface holes. In
Proc. of the International Conference on Image Processing (ICIP’03), pages 903–
906, Barcelona, Spain, September 2003.
[42] Verified Market Research. Immersive technology market size and forecast, 2023.
https://www.verifiedmarketresearch.com/product/immersive-technology-market/.
[43] I. Viola, J. Mulder, F. De Simone, and P. Cesar. Temporal interpolation of dynamic
digital humans using convolutional neural networks. In Proc. of the IEEE International
Conference on Artificial Intelligence and Virtual Reality (AIVR’19), pages
90–97, San Diego, CA, December 2019.
[44] J. Wang and M. Oliveira. Filling holes on locally smooth surfaces reconstructed
from point clouds. Image Vis Comput, 25(1):103–113, January 2007.
[45] J.Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, and L. Shao. Deep 3D human pose
estimation: A review. Computer Vision and Image Understanding, 210:103225:1–
103225:21, September 2021.
[46] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: From
error visibility to structural similarity. IEEE Transactions on Image Processing,
13(4):600–612, April 2004.
[47] X. Wen, T. Li, Z. Han, and Y.-S. Liu. Point cloud completion by skip-attention
network with hierarchical folding. In Proc. of the IEEE/CVF Computer Vision and
Pattern Recognition Conference (CVPR’20), pages 1939–1948, Seattle, WA, June
2020.
[48] C.-H. Wu, C.-F. Hsu, T.-K. Hung, C. Griwodz, W. T. Ooi, and C.-H. Hsu. Quantitative
comparison of point cloud compression algorithms with PCC Arena. IEEE
Transaction on Multimedia, 25:3073–3088, February 2023.
[49] C.-H. Wu, X. Li, R. Rajesh, W. T. Ooi, and C.-H. Hsu. Dynamic 3D point cloud
streaming: Distortion and concealment. In Proc. of the ACM Network and Operating
System Support for Digital Audio and Video (NOSSDAV’21), pages 9–14, Istanbul,
Turkey, September 2021.
[50] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun. GRnet: Gridding residual
network for dense point cloud completion. In Proc. of the European Conference on
Computer Vision (ECCV’20), pages 365–381, Virtual, August 2020.
[51] R. Xu, X. Li, B. Zhou, and C. C. Loy. Deep flow-guided video inpainting. In Proc. of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19),
pages 3723–3732, Long Beach, CA, June 2019.
[52] Y. Xu, X. Tong, and U. Stilla. Voxel-based representation of 3D point clouds: Methods,
applications, and its potential use in the construction industry. Automation in
Construction, 126:103675:1–103675:26, June 2021.
[53] M. Yang, D. Wu, Z. Wang, M. Hu, and Y. Zhou. Understanding and improving perceptual
quality of volumetric video streaming. In Proc. of the IEEE International
Conference on Multimedia and Expo (ICME’23), pages 1979–1984, Brisbane, Australia,
July 2023.
[54] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Generative image inpainting
with contextual attention. In Proc. of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR’18), pages 5505–5514, Salt Lake City, UT, June
2018.
[55] Y. Zeng, Y. Qian, Q. Zhang, J. Hou, Y. Yuan, and Y. He. IDEA-Net: Dynamic 3D
point cloud interpolation via deep embedding alignment. In Proc. of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR’22), pages 6338–
6347, New Orleans, LA, June 2022.
[56] E. Zerman, R. Kulkarni, and A. Smolic. User behavior analysis of volumetric video
in augmented reality. In Proc. of the International Conference on Quality of Multimedia
Experience (QoMEX’21), pages 129–132, Virtual, June 2021.
[57] Q.-Y. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing,
2023. http://www.open3d.org/.