在COVID-19流行期間，對人類生理狀況的遠端估計引起了緊急的關注。本文重點介紹了臉部影像對遠程光體積變化描記圖(rPPG)的估計，並解決大規模基準數據集不足的問題。我們提出了一個端到端的RErPPG-Net，包括一個消除網路和一個嵌入網路，以增強現有的rPPG基準數據集。在提出的增強方法中，消除網路首先消除輸入影像中的任何固有rPPG信號，然後嵌入網路將另一個光體積變化描記圖(PPG)信號嵌入到影像中，以生成攜帶指定PPG信號的增強影像。為了從未成對的影像中訓練模型，我們提出了一種新的雙循環一致性約束，以強制RErPPG-Net學習穩健且準確地消除和嵌入微弱的rPPG信號。新的基準數據集“Aug-rPPG”由UBFC-rPPG和PURE數據集增強而成，包括5776個影像由42個受試者和76個不同的rPPG信號組成。我們的實驗結果表明，現有的rPPG估計器確實受益於增強數據集，並且在新基準數據集進行微調時取得顯著的改進。

Remote estimation of human physiological condition has attracted urgent attention during the pandemic of COVID-19. In this paper, we focus on the estimation of remote photoplethysmography (rPPG) from facial videos and address the deficiency issues of large-scale benchmarking datasets. We propose an end-to-end RErPPG-Net, including a Removal-Net and an Embedding-Net, to augment existing rPPG benchmark datasets. In the proposed augmentation scenario, the Removal-Net will first erase any inherent rPPG signals in the input video and then the Embedding-Net will embed another PPG signal into the video to generate an augmented video carrying the specified PPG signal. To train the model from unpaired videos, we propose a novel double-cycle consistent constraint to enforce the RErPPG-Net to learn to robustly and accurately remove and embed the delicate rPPG signals. The new benchmark "Aug-rPPG dataset" is augmented from UBFC-rPPG and PURE datasets and includes 5776 videos from 42 subjects with 76 different rPPG signals. Our experimental results show that existing rPPG estimators indeed benefit from the augmented dataset and achieve significant improvement when fine-tuned on the new benchmark.

摘要i
Abstract ii
Acknowledgements
1 Introduction 1
2 Related Work 4
2.1 Remote Photoplethysmography Estimation 4
2.2 Data Augmentation 6
3 Method 7
3.1 Overview 7
3.2 RErPPG-Net 8
3.3 Double-Cycle Consistent Learning for Embedding-Net 9
3.4 Double-Cycle Consistent Learning for Removal-Net 11
3.5 Loss function 13
4 Experiments 16
4.1 Datasets 16
4.2 Implementation Details 17
4.3 Evaluation Metrics 17
4.4 Ablation Study 18
4.5 Results and Comparison 22
4.6 Aug-rPPG Dataset 25
4.7 Visualized Examples 25
5 Conclusion 29
References 30

[1] X. Niu, S. Shan, H. Han, and X. Chen, “Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation,” IEEE Transactions on Image Processing, vol. 29, pp. 2409–2423, 2019.
[2] Y.-Y. Tsou, Y.-A. Lee, and C.-T. Hsu, “Multi-task learning for simultaneous video generation and remote photoplethysmography estimation,” in Proceedings of the Asian Conference on Computer Vision, 2020.
[3] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
[4] R. Song, H. Chen, J. Cheng, C. Li, Y. Liu, and X. Chen, “Pulsegan: Learning to generate realistic pulse waveforms in remote photoplethysmography,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1373–1384, 2021.
[5] W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 349–365, 2018.
[6] G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Transactions on Biomedical Engineering, vol. 60, no. 10, pp. 2878–2886, 2013.
[7] X. Niu, Z. Yu, H. Han, X. Li, S. Shan, and G. Zhao, “Video-based remote physiological measurement via cross-verified feature disentangling,” in European Conference on Computer Vision, pp. 295–310, Springer, 2020.
[8] H. Lu, H. Han, and S. K. Zhou, “Dual-gan: Joint bvp and noise modeling for remote physiological measurement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12404–12413, 2021.
[9] X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585, IEEE, 2018.
[10] Y.-Y. Tsou, Y.-A. Lee, C.-T. Hsu, and S.-H. Chang, “Siamese-rppg network: Remote photoplethysmography signal estimation from face videos,” in Proceedings of the 35th annual ACM symposium on applied computing, pp. 2066–2073, 2020.
[11] F. Bousefsaf, A. Pruski, and C. Maaoui, “3d convolutional neural networks for remote pulse rate measurement and mapping from facial video,” Applied Sciences, vol. 9, no. 20, p. 4364, 2019.
[12] E. Lee, E. Chen, and C.-Y. Lee, “Meta-rppg: Remote heart rate estimation using a transductive meta-learner,” in European Conference on Computer Vision, pp. 392–409, Springer, 2020.
[13] R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the british machine vision conference, Newcastle, UK, pp. 3–6, 2018.
[14] Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 151–160, 2019.
[15] X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4264–4271, 2014.
[16] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE transactions on biomedical engineering, vol. 58, no. 1, pp. 7–11, 2010.
[17] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light.,” Optics express, vol. 16, no. 26, pp. 21434–21445, 2008.
[18] W. Wang, A. C. den Brinker, S. Stuijk, and G. De Haan, “Algorithmic principles of remote ppg,” IEEE Transactions on Biomedical Engineering, vol. 64, no. 7, pp. 1479–1491, 2016.
[19] W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE transactions on biomedical engineering, vol. 63, no. 9, pp. 1974–1984, 2015.
[20] Q. Li, Y. Liu, and Z. Sun, “Age progression and regression with spatial attention modules,” in Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11378–11385, 2020.
[21] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro, “Video-tovideo synthesis,” arXiv preprint arXiv:1808.06601, 2018.
[22] D. McDuff, X. Liu, J. Hernandez, E. Wood, and T. Baltrusaitis, “Synthetic data for multiparameter camera-based physiological sensing,” in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3742–3748,
IEEE, 2021.
[23] X. Zhang, R. Ng, and Q. Chen, “Single image reflection separation with perceptual losses,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4786–4794, 2018.
[24] S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, and J. Dubois, “Unsupervised skin tissue segmentation for remote photoplethysmography,” Pattern Recognition Letters,
vol. 124, pp. 82–90, 2019.
[25] R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 1056–1062, IEEE, 2014.
[26] X. Niu, H. Han, S. Shan, and X. Chen, “Vipl-hr: A multi-modal database for pulse estimation from less-constrained face video,” in Asian Conference on Computer Vision, pp. 562–576, Springer, 2018.