帳號:guest(18.223.206.144)          離開系統
字體大小: 字級放大   字級縮小   預設字形  

詳目顯示

以作者查詢圖書館館藏以作者查詢臺灣博碩士論文系統以作者查詢全國書目
作者(中文):陳元捷
作者(外文):Chen, Yuan-Jie
論文名稱(中文):HPFL: 融合多種感測器模式與異質隱私敏感度的聯邦式學習
論文名稱(外文):HPFL: Federated Learning by Fusing Multiple Sensor Modalities with Heterogeneous Privacy Sensitivity Levels
指導教授(中文):徐正炘
指導教授(外文):Hsu, Cheng-Hsin
口試委員(中文):陳健
黃俊穎
口試委員(外文):Chen, Chien
Huang, Chun-Ying
學位類別:碩士
校院名稱:國立清華大學
系所名稱:資訊工程學系
學號:109062467
出版年(民國):111
畢業學年度:110
語文別:中文
論文頁數:59
中文關鍵詞:信息系統多媒體信息系統安全和隱私隱私保護計算方法分類監督學習
外文關鍵詞:Information systemsMultimedia information systemsSecurity and privacyPrivacy protectionsComputing methodologiesSupervised learning by classification
相關次數:
  • 推薦推薦:0
  • 點閱點閱:280
  • 評分評分:*****
  • 下載下載:0
  • 收藏收藏:0
通過多模式感測器資料解決分類問題在近幾年變得流行,包含疾病診斷,安全保障,娛樂遊戲等應用。
考慮到一些高隱私敏感度的感測器,例如攝影機和麥克風,收集資料進行中心化機器學習容易造成隱私洩露風險。
儘管現有的聯合學習 (FL) 演算法允許使用者把高隱私敏感度資料保留在本地參與模型訓練,但是受限於模型分類準確度較低,訓練時間較長等缺點。
在這篇論文中,我們考慮到不同感測器收集的資料有不同的隱私敏感度,提出異質隱私聯合學習算法 (HPFL) 來利用低隱私敏感度資料 (例如毫米波雷達收集的點雲) 中的訊息改善模型性能,同時使高隱私敏感度資料保持在用戶本地。
在我們提出的HPFL中,要求每個使用者把低隱私敏感度的資料上傳到服務器,HPFL使用這些資料對服務器模型進行更進一步的訓練。
用了我們最好的知識, 在相關的FL算法研究中沒有考慮到多模式感測器收集的資料有不同隱私敏感度的性質。
我們使用兩種流行的機器學習應用來測試HPFL演算法的性能,語義分割,和情感識別。%,和進食動作辨認。
我們的實驗結果充分驗證了HPFL的性能。
在語義分割應用中,HPFL在前景準確度上超過FedAvg 18.20\%;在情感識別應用中,HPFL在F1-score上超過FedAvg 4.20\%;這两個實驗都建立在非獨立同分佈的資料分佈上。
與其他相關FL工作比較,HPFL在語義分割上獲得12.40\%--17.70\%性能提升,在情感識別上獲得2.54\%--4.10\%的性能提升。
在收斂速度方面,HPFL相比FedAvg提前24輪達到其最大準確率。
同時,HPFL沒有提升使用者的計算成本,在兩個應用中只略有增加了少量網路通訊成本,相比FedAvg,HPFL消耗額外通訊成本佔比:語義分割5.95\%,情感識別0.15\%。
上述實驗結果證明HPFL在FL多形態應用中減少了非獨立同分佈資料分佈造成的性能損失,並比其他FL演算法結果更好。
HPFL具有很好的擴展性,在未來的工作中,我們會進行收斂性分析,隱私洩露風險分析,HPFL在複雜模型的應用,結合拆分學習(SL),和其他分散式學習方法。
Solving classification problems to understand multi-modality sensor data has become popular,
%with healthcare, security, entertainment, and many other applications. However,
but rich-media sensors, e.g., RGB cameras and microphones, are privacy-invasive. Though existing
%studies with the
Federated Learning (FL) algorithms allow clients to keep their sensor data private, they suffer from degraded performance,
particularly lower classification accuracy and longer training time, when compared to centralized learning.
We propose a Heterogeneous Privacy Federated Learning (HPFL) paradigm to capitalize on the information in the privacy insensitive data (such as mmWave point clouds) while keeping the privacy-sensitive data (such as RGB images) private because sensor data are of diverse sensitivity levels.
We mainly require that each client share privacy insensitive data to a server for fine-tuning the server model, reducing the performance between FL and centralized learning.
{\em To our best knowledge, multiple media/modalities with diverse privacy sensitivity levels have never been considered in the FL setup.} We evaluate the HPFL paradigm on two representative classification problems: (i) semantic segmentation, and (ii) emotion recognition. %, and (iii) food intake actively recognition.
Extensive experiments demonstrate that the HPFL paradigm can:
(i) outperform the popular FedAvg by 18.20\% in foreground accuracy (semantic segmentation) and 4.20\% in F1-score (emotion recognition) under non-i.i.d.~sample distributions,
(ii) surpass the state-of-the-art advanced FL algorithms by 12.40\%--17.70\% in foreground accuracy and 2.54\%--4.10\%,
(iii) achieve FedAvg's maximum foreground accuracy 24 rounds sooner, and
(iv) incur no extra client-side computation overhead and negligible communication overhead of 5.95\% (semantic segmentation) and 0.15\% (emotion recognition).
These results show that HPFL successfully reduce the impact of non-i.i.d.~sample distribution in FL and outperforms the related state-of-the-art FL algorithms in multi-modal applications.
HPFL can be extended in multiple directions in the future, including but not limited to convergence analysis, privacy leakage analysis, complex multi-modal model structure, generation for split learning, and other distributive learning approaches.
Contents

Acknowledgments i
Abstract ii
中文摘要 iii
1 Introduction 1
1.1 Contributions 3
1.2 Limitations 4
1.3 Organizations 4
2 Background 6
2.1 Machine Learning 6
2.2 Distributed Machine Learning 7
2.3 Knowledge Distillation 9
2.4 Multi-modal Representation Learning 10
3 Related Work 12
3.1 Data Sharing in FL 12
3.2 Federated Distillation 13
3.3 Federated Transfer Learning 14
3.4 Advanced FL Algorithms 15
4 Heterogeneous Privacy Federated Learning (HPFL) 16
4.1 System Overview 16
4.2 Notations 18
4.3 Procedure 18
5 Multi-modal Neural Networks and Applications 22
5.1 Semantic Segmentation 23
5.2 Emotion Recognition 23
6 Multi-modal Datasets 25
6.1 MFNet Dataset 26
6.2 LMF Dataset 27
7 Evaluations 28
7.1 Implementations 28
7.2 Hyperparameters 28
7.3 Parameters and Metrics 29

7.4 Parameter Selections 29
7.5 Performance Comparisons 32
8 Conclusion 40
8.1 Concluding Remarks 40
8.2 Future Work 41
Bibliography



Bibliography

[1] Multimodal sensor market in 2022 : Detailed study on business strate- gies, development factors, future trends, opportunities, and demand out- look till 2028 with fastest growing regions and countries data, 2022. https://southeast.newschannelnebraska.com/story/46208517/Multimodal-Sensor- Market.

[2] D. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama. Fed- erated learning based on dynamic regularization. In Proc. of International Confer- ence on Learning Representations (ICLR), pages 1–43, Los Angeles, USA, May 2020.

[3] M. Aledhari, R. Razzak, R. Parizi, and F. Saeed. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access, 8, 2020.

[4] P. Antoniadis, I. Pikoulis, P. Filntisis, and P. Maragos. An audiovisual and contex- tual approach for categorical and continuous emotion recognition in-the-wild. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pages 3645–3651, Virtual, 2021.

[5] I. Ariav and I. Cohen. An end-to-end multimodal voice activity detection using wavenet encoder and residual networks. IEEE Journal of Selected Topics in Signal Processing, 13(2):265–274, 2019.

[6] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov. How to backdoor federated learning. In Proc. of International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2938–2948, Virtual, 2020.

[7] T. Baltrusaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 41(2), 2018.

[8] H. B. Barlow. Unsupervised learning. Neural Computation, 1(3):295–311, 1989.

[9] P. Bellavista, L. Foschini, and A. Mora. Decentralised learning in federated de- ployment environments: A system-level survey. ACM Computing Surveys, 54(1), 2021.
[10] N. Burkart and M. Huber. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245–317, 2021.
[11] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang,
S. Lee, and S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4), 2008.
[12] M. Chen, D. Gunduz, K. Huang, W. Saad, M. Bennis, A. V. Feljan, and H. V. Poor. Distributed learning in wireless networks: Recent progress and future challenges. IEEE Journal on Selected Areas in Communications, 2021.
[13] M. Chen, N. Shlezinger, H. Poor, Y. Eldar, and S. Cui. Communication-efficient federated learning. Proc. of National Academy of Sciences, 118(17), 2021.
[14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for con- trastive learning of visual representations. In Proc. of International Conference on Machine Learning (ICML), pages 1597–1607, Virtual, June 2020.
[15] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4):83– 93, 2020.
[16] Y. Chen, X. Sun, and Y. Jin. Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE Transactions on Neural Networks and Learning Systems, 31(10), 2019.
[17] Z. Chen, L. Zhang, Z. Cao, and J. Guo. Distilling the knowledge from handcrafted features for human activity recognition. IEEE Transactions on Industrial Informat- ics, 14(10):4334–4342, 2018.
[18] M. Cheng, X. Jiao, Y. Liu, M. Shao, X. Yu, Y. Bai, Z. Wang, S. Wang, N. Tuohuti,
S. Liu, et al. Estimation of soil moisture content under high maize canopy coverage from uav multimodal data and machine learning. Agricultural Water Management, 264:107530, 2022.
[19] Y. Cheng, D. Wang, P. Zhou, and T. Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1), 2018.

[20] P. Cunningham, M. Cord, and S. Delany. Supervised learning. In Machine Learn- ing Techniques for Multimedia, pages 21–49. 2008.
[21] B. Darwin, P. Dharmaraj, S. Prince, D. Popescu, and D. Hemanth. Recognition of bloom/yield in crop images using deep learning models for smart agriculture: A review. Agronomy, 11(4), 2021.
[22] V. De Silva, J. Roche, and A. Kondoz. Robust fusion of LiDAR and wide-angle camera data for autonomous mobile robots. Sensors, 18(8):2730, 2018.
[23] F. Demrozi, G. Pravadelli, A. Bihorac, and P. Rashidi. Human activity recognition using inertial, physiological and environmental sensors: A comprehensive survey. IEEE Access, 8:210816–210836, 2020.
[24] C. Dinh, N. Tran, and T. Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
[25] H. Du, P. Henry, X. Ren, M. Cheng, D. Goldman, S. Seitz, and D. Fox. Interactive 3D modeling of indoor environments with a consumer depth camera. In Proc. of International Conference on Ubiquitous Computing (UBIC), pages 75–84, Ankara, Turkey, 2011.
[26] C. Dwork, A. Roth, et al. The algorithmic foundations of differential privacy.
Found. Trends Theor. Comput. Sci., 9(3-4):211–407, 2014.

[27] A. M. Elbir, S. Coleri, and K. V. Mishra. Hybrid federated and centralized learn- ing. In Proc. of European Signal Processing Conference (EUSIPCO), pages 1541– 1545, Dublin, Ireland, 2021.
[28] Z. Erickson, E. Xing, B. Srirangam, S. Chernova, and C. Kemp. Multimodal mate- rial classification for robots using spectroscopy and high resolution texture imag- ing. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10452–10459, Las Vegas, USA, 2020.
[29] A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning: A meta- learning approach. In Proc. of International Conference on Neural Information Processing Systems (NeurIPS), December 2020.
[30] V. Ganchenko and A. Doudkin. Image semantic segmentation based on convolu- tional neural networks for monitoring agricultural vegetation. In Proc. of Inter- national Conference on Pattern Recognition and Information Processing (PRIP), pages 52–63, Minsk, Belarus, 2019.

[31] J. Gao, P. Li, Z. Chen, and J. Zhang. A survey on deep learning for multimodal data fusion. Neural Computation, 32(5):829–864, 2020.

[32] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, P. Martinez- Gonzalez, and J. Garcia-Rodriguez. A survey on deep learning techniques for image and video semantic segmentation. Elsevier Applied Soft Computing, 70, 2018.

[33] C. Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.

[34] N. Gholizadeh and P. Musilek. Distributed learning applications in power systems: A review of methods, gaps, and challenges. Energies, 14(12):3654, 2021.

[35] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Werns- ing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proc. of International Conference on Machine Learning (ICML), pages 201–210, New York, USA, 2016.

[36] J. Gou, B. Yu, S. Maybank, and D. Tao. Knowledge distillation: A survey. Springer International Journal of Computer Vision, 129(6), 2021.

[37] N. Guha, A. Talwalkar, and V. Smith. One-shot federated learning. arXiv preprint arXiv:1902.11175, 2019.

[38] E. Gumuslu, D. Barkana, and H. Kose. Emotion recognition using EEG and phys- iological data for robot-assisted rehabilitation systems. In Proc. of International Conference on Multimodal Interaction (ICMI), pages 379–387, Utrecht, Nether- lands, 2020.

[39] W. Guo, J. Wang, and S. Wang. Deep multimodal representation learning: A sur- vey. IEEE Access, 7, 2019.

[40] A. Gupta, A. Anpalagan, L. Guan, and A. Khwaja. Deep learning for object detec- tion and scene perception in self-driving cars: Survey, challenges, and open issues. Array, 10:100057, 2021.

[41] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada. MFNet: Towards real- time semantic segmentation for autonomous vehicles with multi-spectral scenes. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5108–5115, British Columbia, Canada, 2017.

[42] L. Hao, N. Rohani, R. Zhao, E. Pulver, H. Mak, O. Kelada, H. Ko, H. Fleming,
F. Gertler, and S. Bhatia. Microenvironment-triggered multimodal precision diag- nostics. Nature Materials, 20(10):1440–1448, 2021.
[43] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein,
H. Eichner, C. Kiddon, and D. Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
[44] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In Proc. of NIPS Deep Learning and Representation Learning Workshop (NeurIPS), December 2015.
[45] W. Hong, X. Luo, Z. Zhao, M. Peng, and T. Quek. Optimal design of hybrid federated and centralized learning in the mobile edge computing systems. In Proc. of IEEE International Conference on Communications Workshops (ICC), pages 1– 6, Virtual, 2021.
[46] F. Huang, X. Zhang, Z. Zhao, J. Xu, and Z. Li. Image–text sentiment analysis via deep multimodal attentive fusion. Knowledge-Based Systems, 167:26–37, 2019.
[47] L. Huang, Y. Yin, Z. Fu, S. Zhang, H. Deng, and D. Liu. LoAdaBoost: Loss-based AdaBoost federated machine learning with reduced computational complexity on iid and non-iid intensive care data. Plos One, 15(4), 2020.
[48] L.-R. Jacome-Galarza. Crop yield prediction utilizing multimodal deep learning. In Proc. of Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6, Chaves, Portugal, 2021.
[49] C. Janiesch, P. Zschech, and K. Heinrich. Machine learning and deep learning.
Electronic Markets, 31(3):685–695, 2021.

[50] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim. Communication- efficient on-device machine learning: Federated distillation and augmentation un- der non-iid private data. In Proc. of International Conference on Neural Informa- tion Processing Systems (NeurIPS), December 2018.
[51] E. Jeong, S. Oh, J. Park, H. Kim, M. Bennis, and S.-L. Kim. Hiding in the crowd: Federated data augmentation for on-device learning. IEEE Intelligent Systems, 36(5), 2020.
[52] H. Jiang and Y. Guo. Multi-class multimodal semantic segmentation with an im- proved 3D fully convolutional networks. Neurocomputing, 391:220–226, 2020.

[53] M. Jordan and T. Mitchell. Machine learning: Trends, perspectives, and prospects.
Science, 349(6245):255–260, 2015.

[54] L. Kaelbling, M. Littman, and A. Moore. Reinforcement learning: A survey. Jour- nal of Artificial Intelligence Research, 4:237–285, 1996.

[55] P. Kairouz, H. McMahan, B. Avent, A. Bellet, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1–2):1– 210, 2021.

[56] D. Kothandaraman, A. Nambiar, and A. Mittal. Domain adaptive knowledge dis- tillation for driving scene semantic segmentation. In Proc. of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 134–143, Virtual, 2021.

[57] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

[58] V. Kulkarni, M. Kulkarni, and A. Pant. Survey of personalization techniques for federated learning. In Proc. of Fourth World Conference on Smart Trends in Sys- tems, Security and Sustainability (WorldS4), pages 794–797, Virtual, 2020.

[59] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. of IEEE, 86(11):2278–2324, 1998.

[60] J. Lee, S. Kim, S. Kim, J. Park, and K. Sohn. Context-aware emotion recognition networks. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pages 10143–10152, Seoul, Korea, 2019.

[61] D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation.
arXiv preprint arXiv:1910.03581, 2019.

[62] H. Li, A. Shrestha, H. Heidari, J. Le Kernec, and F. Fioranelli. Bi-lstm network for multimodal continuous human activity recognition and fall detection. IEEE Sensors Journal, 20(3):1191–1201, 2019.

[63] M. Li, D. Andersen, J. Park, A. Smola, A. Ahmed, V. Josifovski, J. Long,
E. Shekita, and B.-Y. Su. Scaling distributed machine learning with the param- eter server. In Proc. of USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 583–598, Broomfield, USA, 2014.

[64] Q. Li, B. He, and D. Song. Model-contrastive federated learning. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10713–10722, Virtual, 2021.
[65] T. Li, A. Sahu, A. Talwalkar, and V. Smith. Federated learning: challenges, meth- ods, and future directions. IEEE Signal Processing Magazine, 37(3), 2020.
[66] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In Proc. of Conference on Systems and Machine Learning (SysML), 2:429–450, 2018.
[67] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
[68] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou. FedBN: Federated learning on non-iid features via local batch normalization. In Proc. of International Conference on Learning Representations (ICLR), pages 1–27, Virtual, 2020.
[69] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional instance-aware semantic segmentation. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2359–2367, Honolulu, Hawaii, 2017.
[70] T. Lin, L. Kong, S. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. In Proc. of Conference on Neural Information Pro- cessing Systems (NeurIPS), 2020.
[71] Q. Liu, C. Chen, J. Qin, Q. Dou, and P.-A. Heng. FedDG: Federated domain gen- eralization on medical image segmentation via episodic learning in continuous fre- quency space. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1013–1023, Virtual, 2021.
[72] X. Liu, Y. Han, S. Bai, Y. Ge, T. Wang, X. Han, S. Li, J. You, and J. Lu. Importance- aware semantic segmentation in self-driving with discrete wasserstein training. In Proc. of AAAI Conference on Artificial Intelligence (AAAI), volume 34, pages 11629–11636, Hilton, USA, 2020.
[73] Y. Liu, Y. Kang, C. Xing, T. Chen, and Q. Yang. A secure federated transfer learning framework. IEEE Intelligent Systems, 35(4):70–82, 2020.
[74] Y. Liu, K. Wang, G. Li, and L. Lin. Semantics-aware adaptive knowledge distilla- tion for sensor-to-vision action recognition. IEEE Transactions on Image Process- ing, 30:5573–5588, 2021.

[75] Z. Liu, Y. Shen, V. Lakshminarasimhan, P. Liang, A. Zadeh, and L.-P. Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064, 2018.
[76] Y. Lu, K. Xu, L. Zhang, M. Deguchi, H. Shishido, T. Arie, R. Pan, A. Hayashi,
L. Shen, S. Akita, et al. Multimodal plant healthcare flexible sensor system. ACS Nano, 14(9):10966–10975, 2020.
[77] M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34, 2021.
[78] L. Lyu, H. Yu, and Q. Yang. Threats to federated learning: A survey. arXiv preprint arXiv:2003.02133, 2020.
[79] N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge- Based Systems, 161:124–133, 2018.
[80] Y. Matsubara, M. Levorato, and F. Restuccia. Split computing and early exiting for deep learning applications: Survey and research challenges. arXiv preprint arXiv:2103.04505, 2021.
[81] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Arcas. Communication- efficient learning of deep networks from decentralized data. In Proc. of Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), pages 1273– 1282, Lauderdale, 2017.
[82] J. Mills, J. Hu, and G. Min. Communication-efficient federated learning for wire- less edge intelligence in IoT. IEEE Internet of Things Journal, 7(7):5986–5994, 2019.
[83] G. Muhammad, F. Alshehri, F. Karray, A. El Saddik, M. Alsulaiman, and T. Falk. A comprehensive survey on multimodal medical signals fusion for smart healthcare systems. Information Fusion, 76:355–375, 2021.
[84] H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic seg- mentation. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pages 1520–1528, Santiago, Chile, 2015.
[85] B. Nojavanasghari, D. Gopinath, J. Koushik, T. Baltruvsaitis, and L.-P. Morency. Deep multimodal fusion for persuasiveness prediction. In Proc. of ACM Inter-

national Conference on Multimodal Interaction (ICMI), pages 284–288, Tokyo, Japan, 2016.
[86] T. Ogawa, Y. Sasaka, K. Maeda, and M. Haseyama. Favorite video classification based on multimodal bidirectional LSTM. IEEE Access, 6:61401–61409, 2018.
[87] J. Park, S. Wang, A. Elgabli, S. Oh, E. Jeong, H. Cha, H. Kim, S.-L. Kim, and
M. Bennis. Distilling on-device intelligence at the network edge. arXiv preprint arXiv:1908.05895, 2019.
[88] A. Pemasiri, K. Nguyen, S. Sridharan, and C. Fookes. Multi-modal semantic image segmentation. Computer Vision and Image Understanding, 202:103085, 2021.
[89] F. Pokorny, M. Fivser, F. Graf, P. Marschik, and B. Schuller. Sound and the city: Current perspectives on acoustic geo-sensing in urban environment. Acta Acustica united with Acustica, 105(5), 2019.
[90] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508, 2018.
[91] B. Rajalingam and R. Priya. Hybrid multimodality medical image fusion technique for feature enhancement in medical diagnosis. International Journal of Engineer- ing Science Invention, 2(Special issue):52–60, 2018.
[92] N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. Lanckriet, R. Levy, and
N. Vasconcelos. A new approach to cross-modal multimedia retrieval. In Proc. of ACM International Conference on Multimedia (MM), pages 251–260, Firenze, Italy, 2010.
[93] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konecny, S. Kumar, and
H. McMahan. Adaptive federated optimization. In Proc. of International Confer- ence on Learning Representations (ICLR), pages 1–38, Virtual, 2021.
[94] G. Roque and V. Padilla. LPWAN based IoT surveillance system for outdoor fire detection. IEEE Access, 8, 2020.
[95] M. Saradjian and M. Akhoondzadeh. Thermal anomalies detection before strong earthquakes using interquartile, wavelet and kalman filter methods. Natural Haz- ards and Earth System Sciences, 11(4):1099–1108, 2011.
[96] I. Sarker. Machine learning: Algorithms, real-world applications and research di- rections. SN Computer Science, 2(3):1–21, 2021.

[97] F. Sattler, S. Wiedemann, K.-R. Mu¨ller, and W. Samek. Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Transac- tions on Neural Networks and Learning Systems, 31(9), 2020.
[98] A. Saxena, A. Khanna, and D. Gupta. Emotion recognition and detection methods: A comprehensive survey. Journal of Artificial Intelligence and Systems, 2(1), 2020.
[99] H. Seo, J. Park, S. Oh, M. Bennis, and S.-L. Kim. Federated knowledge distillation.
arXiv preprint arXiv:2011.02367, 2020.

[100] S. Sharma, C. Xing, Y. Liu, and Y. Kang. Secure and efficient federated transfer learning. In Proc. of IEEE International Conference on Big Data (Big Data), pages 2569–2576, Los Angeles, CA, 2019.
[101] G. Shen, X. Wang, X. Duan, H. Li, and W. Zhu. MEmoR: a dataset for multi- modal emotion reasoning in videos. In Proc. of ACM International Conference on Multimedia (MM), pages 493–502, Seattle, USA, 2020.
[102] K. Shih, S. Singh, and D. Hoiem. Where to look: Focus regions for visual question answering. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4613–4621, Las Vegas, USA, 2016.
[103] U. Sulubacak, O. Caglayan, S.-A. Gronroos, A. Rouhe, D. Elliott, L. Specia, and
J. Tiedemann. Multimodal machine translation through visuals and speech. Ma- chine Translation, 34(2):97–147, 2020.
[104] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu. A survey on deep trans- fer learning. In Proc. of International Conference on Artificial Neural Networks (ICANN), pages 270–279, Rhodes, Greece, 2018.
[105] J. Tang, R. Shivanna, Z. Zhao, D. Lin, A. Singh, E. Chi, and S. Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020.
[106] Z. Tang, S. Shi, X. Chu, W. Wang, and B. Li. Communication-efficient distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307, 2020.
[107] C. Teague, J. Heller, B. Nevius, A. Carek, S. Mabrouk, F. Garcia-Vicente, O. Inan, and M. Etemadi. A wearable, multimodal sensing system to monitor knee joint health. IEEE Sensors Journal, 20(18):10323–10334, 2020.
[108] C. Thapa, M. Chamikara, S. Camtepe, and L. Sun. Splitfed: When federated learn- ing meets split learning. arXiv preprint arXiv:2004.12088, 2020.

[109] H. Tian, Y. Tao, S. Pouyanfar, S.-C. Chen, and M.-L. Shyu. Multimodal deep rep- resentation learning for video classification. World Wide Web, 22(3):1325–1341, 2019.
[110] M. Treml, J. Arjona-Medina, T. Unterthiner, et al. Speeding up semantic seg- mentation for autonomous driving. In Proc. of Conference on Neural Information Processing Systems (NeurIPS), 2016.
[111] J. Tu, H. Li, X. Yan, M. Ren, Y. Chen, M. Liang, E. Bitar, E. Yumer, and R. Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving. arXiv preprint arXiv:2101.06784, 2021.
[112] A. Valada, R. Mohan, and W. Burgard. Self-supervised model adaptation for multimodal semantic segmentation. International Journal of Computer Vision, 128(5):1239–1285, 2020.
[113] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018.
[114] I. Wagner and D. Eckhoff. Technical privacy metrics: a systematic survey. ACM Computing Surveys (CSUR), 51(3):1–38, 2018.
[115] H. Wang, P. Cai, R. Fan, Y. Sun, and M. Liu. End-to-end interactive prediction and planning with optical flow distillation for autonomous driving. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2229–2238, Virtual, 2021.
[116] H. Wang, L. Munoz-Gonzalez, D. Eklund, and S. Raza. Non-iid data re-balancing at IoT edge with peer-to-peer federated learning for anomaly detection. In Proc. of ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec), pages 153–163, Virtual, 2021.
[117] H. Wang, D. Zhang, Y. Song, S. Liu, Y. Wang, D. Feng, H. Peng, and W. Cai. Seg- menting neuronal structure in 3D optical microscope images via knowledge distil- lation with teacher-student network. In Proc. of IEEE International Symposium on Biomedical Imaging (ISBI), pages 228–231, Venice, Italy, 2019.
[118] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. Poor. Tackling the objective inconsis- tency problem in heterogeneous federated optimization. In Proc. of International Conerence on Neural Information Processing Systems (NeurIPS), 33:7611–7623, 2020.

[119] L. Wang, W. Wang, and B. Li. CMFL: Mitigating communication overhead for federated learning. In Proc. of IEEE International Conference on Distributed Com- puting Systems (ICDCS), pages 954–964, Dallas, USA, 2019.
[120] L. Wang and K.-J. Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6), 2021.
[121] Y. Wang. Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion. ACM Transactions on Multimedia Computing, Communications, and Ap- plications, 17(1s), 2021.
[122] K. Wei, J. Li, M. Ding, C. Ma, H. Yang, F. Farokhi, S. Jin, T. Quek, and H. Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454–3469, 2020.
[123] J. Wu, Q. Liu, Z. Huang, Y. Ning, H. Wang, E. Chen, J. Yi, and B. Zhou. Hi- erarchical personalized federated learning for user modeling. In Proc. of the Web Conference, pages 957–968, Ljubljana, Slovenia, 2021.
[124] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[125] H. Xu and K. Saenko. Ask, attend and answer: Exploring question-guided spa- tial attention for visual question answering. In Proc. of European Conference on Computer Vision (ECCV), pages 451–466, 2016.
[126] M. Xu, X. Wang, X. Zhang, G. Bin, Z. Jia, and K. Chen. Computation-efficient multi-model deep neural network for sleep stage classification. In Proc. of Asia Ser- vice Sciences and Software Engineering Conference (ASSE), pages 1–8, Nagoya, Japan, 2020.
[127] D. Yang, Z. Xu, W. Li, et al. Federated semi-supervised learning for COVID region segmentation in chest CT using multi-national data from China, Italy, Japan. Medical Image Analysis, 70, 2021.
[128] H. Yang, H. He, W. Zhang, and X. Cao. Fedsteg: A federated transfer learning framework for secure image steganalysis. IEEE Transactions on Network Science and Engineering, 8(2):1084–1094, 2020.
[129] X. Yin, Y. Zhu, and J. Hu. A comprehensive survey of privacy-preserving federated learning: A taxonomy, review, and future directions. ACM Computing Surveys (CSUR), 54(6):1–36, 2021.

[130] N. Yoshida, T. Nishio, M. Morikura, K. Yamamoto, and R. Yonetani. Hybrid- FL for wireless networks: Cooperative learning mechanism using non-iid data. In Proc. of IEEE International Conference on Communications (ICC), pages 1–7, Virtual, 2020.
[131] G. Yuan, X. Liu, Q. Yan, S. Qiao, Z. Wang, and L. Yuan. Hand gesture recogni- tion using deep feature fusion network based on wearable sensors. IEEE Sensors Journal, 21(1), 2020.
[132] B. Yuhas, M. Goldstein, and T. Sejnowski. Integration of acoustic and visual speech signals using neural networks. IEEE Communications Magazine, 27(11):65–71, 1989.
[133] S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016.
[134] C. Zhang, Z. Yang, X. He, and L. Deng. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing, 14(3):478–493, 2020.
[135] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia. ICNet for real-time semantic segmen- tation on high-resolution images. In Proc. of European Conference on Computer Vision (ECCV), pages 405–420, Munich, Germany, 2018.
[136] R. Zhao, Y. Chen, Y. Wang, Y. Shi, and Z. Xue. An efficient and lightweight approach for intrusion detection based on knowledge distillation. In Proc. of IEEE International Conference on Communications (ICC), pages 1–6, Virtual, 2021.
[137] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
[138] B. Zhou, Y. Tian, S. Sukhbaatar, A. Szlam, and R. Fergus. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167, 2015.
[139] X. Zhou, W. Liang, I. Kevin, K. Wang, H. Wang, L. Yang, and Q. Jin. Deep- learning-enhanced human activity recognition for internet of healthcare things. IEEE Internet of Things Journal, 7(7), 2020.
[140] H. Zhu, J. Xu, S. Liu, and Y. Jin. Federated learning on non-iid data: A survey.
Neurocomputing, 465:371–390, 2021.

[141] X. Zhu and A. Goldberg. Introduction to semi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1):1–130, 2009.
[142] Z. Zhu, J. Hong, and J. Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Proc. of International Conference on Machine Learning (ICML), pages 12878–12889, Virtual, 2021.
 
 
 
 
第一頁 上一頁 下一頁 最後一頁 top
* *