聯邦學習(FL) 被認為是一個迅速發展的研究領域，其中模型在不共享客戶數據的情況下，在一個大規模分佈的客戶端上進行訓練，並由參數伺服器（PS）進行協調。本文探討了一類在聯邦學習應用中普遍存在但具有挑戰性的非凸非平滑損失函數的聯邦學習問題，這些問題由於其複雜的非凸性和非平滑性質以及對通信效率和隱私保護之間的衝突要求，導致其難以處理。

在本論文中，我們提出了一種新穎的聯邦原始-對偶算法，該算法針對非凸非平滑聯邦學習問題進行了定制，並採用了雙向模型稀疏化技術。同時，我們應用了差分隱私技術來確保隱私保護。該算法的獨特見解和特性，以及一些隱私和收斂分析，也被呈現作為聯邦學習算法設計的準則。最後，我們在真實世界數據上進行了大量實驗，以展示所提出算法的有效性以及優於某些最新聯邦學習算法的性能，從而驗證了本研究中呈現的所有分析結果和算法特性。

Federated learning (FL) has been recognized as a rapidly growing research area, where the model is trained over massively distributed clients under the orchestration of a parameter server (PS) without sharing clients' data. This thesis delves into a class of federated problems characterized by non-convex and non-smooth (NCNS) loss functions, that are prevalent in FL applications but challenging to handle due to their intricate non-convexity and non-smoothness nature and the conflicting requirements on communication efficiency and privacy protection.
In this thesis, we propose a novel federated primal-dual algorithm with bidirectional model sparsification tailored for NCNS FL problems, and differential privacy is applied for privacy guarantee. Its unique insightful properties and some privacy and convergence analyses are also presented as the FL algorithm design guidelines. Extensive experiments on real-world data are conducted to demonstrate the effectiveness of the proposed algorithm and much superior performance than some state-of-the-art FL algorithms, together with the validation of all the analytical results and properties.

Abstract (Chinese) I
Abstract II
Acknowledgements (Chinese) III
Table of Contents IV
List of Notations VI
1 Introduction 1
2 Preliminaries of Federated Learning and Differential Privacy 4
2.1 Federated Learning . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Differential Privacy . . . . . . . . . . . . . . . . . . . . . . 5
3 Proposed Privacy-preserving Federated Primal-dual Method 8
3.1 Proposed Primal-dual Method in FL . . . . . . . . . . . .. . . . 8
3.2 DP-FedPDM with Bidirectional Model Sparsification . . . . . . . 11
4 Privacy and Convergence Analysis for DP-FedPDM 16
4.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Privacy Analysis . . . . . . . . . . . . . . .. . . . . . . . . 17
4.3 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . 18
IV
5 Experimental Results and Discussions 21
5.1 Experimental Model . . . . . . . . . . . . . . .. . . . . . . . 21
5.2 Datasets and Benchmark Algorithms . . . . . . . . . . . . . . . 22
5.2.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.2.2 Benchmark Algorithms . . . . . . . . . . . . . . . . . . . . 23
5.3 Parameter Setting . . . . . . . . . . . . . . . . . . . . . . . 23
5.4 Experiment Results . . . . . . . . . . . . . . . .. . . . . . . 23
5.4.1 Impact of model sparsification . . . . . . . . . . . . .. . . 24
5.4.2 Impact of DP . . . . . . . . . . . . . . . . . . . . . . . . 25
5.4.3 Impact of non-convex loss function . . . . . . . . . . . . . 26
5.4.4 Impact of non-smooth regularizer . . . . . . . . . . . . . . 27
5.4.5 Performance comparison with benchmark algorithms . . . . 28
5.4.6 Discussion on communication complexity . . . . . . . . . . . 29
6 Conclusions 32
A Proof of (3.6) 34
B Proof of Theorem 1 35
C Proof of Theorem 3 37
D Proof of Key Lemmas 40
D.1 Proof of Lemma 2 . . . . . . . . . . . . .. . . . . . . . . . . 40
D.2 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . 43
Bibliography 47
Publication List of The Author 52

[1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y Arcas,
“Communication-efficient learning of deep networks from decentralized data,”
in Proc. Artificial Intelligence and Statistics, 2017, pp. 1273–1282.
[2] W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang,
D. Niyato, and C. Miao, “Federated learning in mobile edge networks: A
comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 22,
no. 3, pp. 2031–2063, 2020.
[3] Q. Wu, K. He, and X. Chen, “Personalized federated learning for intelligent
IoT applications: A cloud-edge based framework,” IEEE Open Journal of the
Computer Society, vol. 1, pp. 35–44, 2020.
[4] A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “A survey on
federated learning for resource-constrained IoT devices,” IEEE Internet of
Things Journal, vol. 9, no. 1, pp. 1–24, 2021.
[5] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. V.
Poor, “Federated learning for internet of things: A comprehensive survey,”
IEEE Communications Surveys & Tutorials, vol. 23, no. 3, pp. 1622–1658,
2021.
[6] R. Saha, S. Misra, and P. K. Deb, “FogFL: Fog-assisted federated learning for
resource-constrained IoT devices,” IEEE Internet of Things Journal, vol. 8,
no. 10, pp. 8456–8463, 2020.
[7] P. Kairouz, H. B. McMahan, B. Avent, Bellet et al., “Advances and open
problems in federated learning,” Foundations and Trends in Machine Learning,
vol. 14, no. 1–2, pp. 1–210, 2021.
[8] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence
of FedAvg on non-IID data,” in Proc. International Conference on Learning
Representations (ICLR), 2020, pp. 1–26.
[9] Y. Li, T.-H. Chang, and C.-Y. Chi, “Secure federated averaging algorithm
with differential privacy,” in Proc. IEEE International Workshop on Machine
Learning for Signal Processing (MLSP), 2020, pp. 1–6.
[10] Y. Li, S. Wang, T.-H. Chang, and C.-Y. Chi, “Federated stochastic primaldual
learning with differential privacy,” arXiv preprint arXiv:2204.12284,
2022.
[11] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that
exploit confidence information and basic countermeasures,” in Proc. ACM
SIGSAC Conference on Computer and Communications Security, 2015, pp.
1322–1333.
[12] J. Geiping, H. Bauermeister, H. Dr¨oge, and M. Moeller, “Inverting gradientshow
easy is it to break privacy in federated learning?” in Proc. Advances in
Neural Information Processing Systems (NIPS), 2020, pp. 16 937–16 947.
[13] C. Dwork, A. Roth et al., “The algorithmic foundations of differential privacy,”
Foundations and Trends® in Theoretical Computer Science, vol. 9,
no. 3–4, pp. 211–407, 2014.
[14] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to
backdoor federated learning,” in Proc. International Conference on Artificial
Intelligence and Statistics, 2020, pp. 2938–2948.
[15] Q. Xia, W. Ye, Z. Tao, J. Wu, and Q. Li, “A survey of federated learning for
edge computing: Research problems and solutions,” High-Confidence Computing,
vol. 1, no. 1, pp. 1–13, 2021.
[16] S. Wang, Y. Xu, Z. Wang, T.-H. Chang, T. Q. Quek, and D. Sun, “Beyond
ADMM: A unified client-variance-reduced adaptive federated learning
framework,” in Proc. AAAI Conference on Artificial Intelligence, 2023, pp.
10 175–10 183.
[17] Y. Wang, Q. Shi, and T.-H. Chang, “Why batch normalization damage federated
learning on non-iid data?” IEEE Trans. Neural Networks and Learning
Systems, pp. 1–15, 2023.
[18] X. Liang, S. Shen, J. Liu, Z. Pan, E. Chen, and Y. Cheng, “Variance
reduced local SGD with lower communication complexity,” arXiv preprint
arXiv:1912.12844, 2019.
[19] Y. Wang, Y. Xu, Q. Shi, and T.-H. Chang, “Quantized federated learning
under transmission delay and outage constraints,” IEEE Journal on Selected
Areas in Communications, vol. 40, no. 1, pp. 323–341, 2021.
[20] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani,
“FedPAQ: A communication-efficient federated learning method with periodic
averaging and quantization,” in Proc. International Conference on Artificial
Intelligence and Statistics, 2020, pp. 2021–2031.
[21] F. Sattler, S. Wiedemann, K.-R. M¨uller, and W. Samek, “Robust and
communication-efficient federated learning from non-iid data,” IEEE Trans.
Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3400–3413, 2019.
[22] R. Hu, Y. Gong, and Y. Guo, “Federated learning with sparsified model perturbation:
Improving accuracy under client-level differential privacy,” arXiv
preprint arXiv:2202.07178, 2022.
[23] M. Hong, Z.-Q. Luo, and M. Razaviyayn, “Convergence analysis of alternating
direction method of multipliers for a family of nonconvex problems,” SIAM
Journal on Optimization, vol. 26, no. 1, pp. 337–364, 2016.
[24] D. Hajinezhad, M. Hong, T. Zhao, and Z. Wang, “NESTT: A nonconvex
primal-dual splitting method for distributed and stochastic optimization,” in
Proc. Advances in Neural Information Processing Systems (NIPS), 2016, pp.
3207–3215.
[25] Y. Li, C.-W. Huang, S. Wang, C.-Y. Chi, and Q. S. T. Quek, “Privacypreserving
federated primal-dual learning for non-convex problems with nonsmooth
regularization,” in Proc. IEEE International Workshop on Machine
Learning for Signal Processing (MLSP), 2023, pp. 1–6.
[26] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh,
“Scaffold: Stochastic controlled averaging for federated learning,” in Proc.
International Conference on Machine Learning (ICML), 2020, pp. 5132–5143.
[27] X. Zhang, M. Hong, S. Dhople, W. Yin, and Y. Liu, “FedPD: A federated
learning framework with adaptivity to non-IID data,” IEEE Trans. Signal
Processing, vol. 69, pp. 6055–6070, 2021.
[28] S. Zhou and G. Y. Li, “Federated learning via inexact ADMM,” IEEE Trans.
Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9699–9708,
2023.
[29] J. Ding, S. M. Errapotu, H. Zhang, Y. Gong, M. Pan, and Z. Han, “Stochastic
ADMM based distributed machine learning with differential privacy,” in
Proc. International Conference on Security and Privacy in Communication
Systems, 2019, pp. 257–277.
[30] Y. Li, S. Wang, C.-Y. Chi, and T. Q. Quek, “Differentially private federated
learning in edge networks: The perspective of noise reduction,” IEEE
Network, vol. 36, no. 5, pp. 167–172, 2022.
[31] S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “LDP-Fed: Federated
learning with local differential privacy,” in Proc. ACM International
Workshop on Edge Systems, Analytics and Networking, 2020, pp. 61–66.
[32] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially
private recurrent language models,” in Proc. International Conference
on Learning Representations (ICLR), 2018, pp. 1–14.
[33] ´U. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, and
A. Thakurta, “Amplification by shuffling: From local to central differential
privacy via anonymity,” in Proc. the Thirtieth Annual ACM-SIAM Symposium
on Discrete Algorithms, 2019, pp. 2468–2479.
[34] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning:
A client level perspective,” arXiv preprint arXiv:1712.07557, 2017.
[35] D. K. Dennis, T. Li, and V. Smith, “Heterogeneity for the win: One-shot
federated clustering,” in Proc. International Conference on Machine Learning
(ICML), 2021, pp. 2611–2620.
[36] N. Wang, X. Xiao, Y. Yang, J. Zhao, S. C. Hui, H. Shin, J. Shin, and G. Yu,
“Collecting and analyzing multidimensional data with local differential privacy,”
in Proc. IEEE 35th International Conference on Data Engineering
(ICDE), 2019, pp. 638–649.
[37] X. Li, Y. Chen, C.Wang, and C. Shen, “When deep learning meets differential
privacy: Privacy, security, and more,” IEEE Network, vol. 35, no. 6, pp. 148–
155, 2021.
[38] Y. Li, S.Wang, C.-Y. Chi, and T. Q. S. Quek, “Differentially private federated
clustering over non-iid data,” IEEE Internet of Things Journal, pp. 1–16,
2023.
[39] B. Balle, G. Barthe, and M. Gaboardi, “Privacy amplification by subsampling:
Tight analyses via couplings and divergences,” in Proc. ACM Neural
Information Processing Systems (NIPS), 2018, pp. 6277–6287.
[40] J.-H. Ahn, M. Bennis, and J. Kang, “Model compression via pattern shared
sparsification in analog federated learning under communication constraints,”
IEEE Trans. Green Communications and Networking, vol. 7, no. 1, pp. 298–
312, 2022.
[41] M. Hong and T.-H. Chang, “Stochastic proximal gradient consensus over random
networks,” IEEE Trans. Signal Processing, vol. 65, pp. 2933–2948, 2017.
[42] F. Chen, M. Luo, Z. Dong, Z. Li, and X. He, “Federated metalearning
with fast convergence and efficient communication,” arXiv preprint
arXiv:1802.07876, 2018.
[43] C.-Y. Chi, W.-C. Li, and C.-H. Lin, Convex Optimization for Signal Processing
and Communications: From Fundamentals to Applications. CRC Press,
Boca Raton, FL, Feb. 2017.
[44] N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends in
Optimization, vol. 1, no. 3, pp. 127–239, 2014.
[45] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and
L. Zhang, “Deep learning with differential privacy,” in Proc. ACM SIGSAC
Conference on Computer and Communications Security, 2016, pp. 308–318.
[46] L. Zheng, Y. Liu, X. Xu, C. Chen, W. Sun, X. Hu, L. Wang, and L. Wang,
“Fedpse: Personalized sparsification with element-wise aggregation for federated
learning,” in Proc. International Conference on Learning Representations
(ICLR), 2023, pp. 1–25.
[47] M. Noble, A. Bellet, and A. Dieuleveut, “Differentially private federated learning
on heterogeneous data,” in Proc. International Conference on Artificial
Intelligence and Statistics, 2022, pp. 10 110–10 145.
[48] Q. Tran Dinh, N. H. Pham, D. Phan, and L. Nguyen, “FedDR–randomized
Douglas-Rachford splitting algorithms for nonconvex federated composite optimization,”
in Proc. Advances in Neural Information Processing Systems
(NIPS), 2021, pp. 30 326–30 338.
[49] C. L. Blake and C. J. Merz, “UCI repository of machine learning
databases,” 1998, Irvine. CA: University of California, Department of
Information and Computer Science. [Online]. Available: http://www.ics.uci.
edu/rvmlearnIMLRepository.html.
[50] Y. LeCun, C. Cortes, and C. Burges. The MNIST database. [Online].
Available: http://yann.lecun.com/exdb/mnist.
[51] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated
optimization in heterogeneous networks,” in Proc. Machine Learning
and Systems, 2020, pp. 429–450.
[52] A. E. Durmus, Z. Yue, M. Ramon, M. Matthew, W. Paul, and S. Venkatesh,
“Federated learning based on dynamic regularization,” in Proc. International
Conference on Learning Representations (ICLR), 2021, pp. 1–36.