近年來，機器學習正在逐漸融入在我們的生活中，有許多產品利用機器學習來最佳化他們的服務。為了提供更佳的使用者體驗，我們需要收集更多的資料來提高模型的精準度，而終端設備像是智慧型手機、個人電腦或是 IoT 等等的裝置都是很適合用來收集資料。然而，我們在收集資料的同時，必須保護使用者的資料隱私，而聯邦式學習就是被提出來解決這個問題的一種機器學習設置。使用聯邦式學習，機器學習的模型會在這些裝置上使用本地端收集到的資料進行訓練，因此不會有任何的原始數據被傳輸到網路上，藉此避免資料的直接外流。然而，目前的機器學習框架，像是 TensorFlow 以及 PyTorch 等等的框架，並不是設計給聯邦式學習使用，因此使用者要實作出聯邦式學習的模型需要很多突破這些框架限制的技術，然而這會影響到對於聯邦式學習的研究開發。為了解決這個問題，我們提出一個聯邦式學習的應用框架，UNUS。UNUS 具有高效能的特性，並且完整實做了底層的計算、溝通、容錯機制等等，並且提供高階的 API 讓使用者可以快速套用並自定義聯邦式學習的演算法，藉此加速聯邦式學習的研究與開發。同時，UNUS 也實作了 Stale Synchronous Parallel，使得框架可以同時提供同步與非同步的訓練機制。

In recent years, machine learning has integrated into our life deeply. Lots of products take advantage of machine learning to optimize their services. To provide better user experience, collecting more data is necessary, and terminal equipment like smartphones, PC, and IoT devices are good to go. However, we have to pay great attention to the data privacy of users while collecting the data. As a result, Federated Learning is proposed to deal with this kind of problem. With Federated Learning, models will be trained on the local device with only the local data. No data will be uploaded to the server so that direct data leakage can be prevented. Nevertheless, current ML/DL frameworks, such as TensorFlow and PyTorch, are not designed for FL. It takes lots of hacking to build a Federated Learning model with existing machine learning frameworks, which is not good for research of Federated Learning. Thus, we propose UNUS, a high-performance Federated Learning framework, which provides complete infrastructures for Federated Learning. UNUS handles the loss of messages and clients during the training of Federated Learning, and the convenience that comes from custom API makes the development FL simpler and faster. Also, UNUS is integrated with stale synchronous, which provides both synchronous and asynchronous mechanisms.

1 Introduction ................................................... 1
2 Related Work ................................................... 3
2.1 Algorithms ................................................... 3
2.2 Frameworks ................................................... 5
3 Design Goal .................................................... 8
4 System Design & Implementation ................................. 10
4.1 Execution Model .............................................. 10
4.1.1 Workflow ................................................... 10
4.1.2 Metadata ................................................... 11
4.1.3 Modules .................................................... 12
4.1.4 Synchronization Model ...................................... 14
4.2 Implementation Details ....................................... 14
4.2.1 Registration ............................................... 15
4.2.2 Communication .............................................. 15
4.2.3 Synchronization ............................................ 16
4.2.4 Model Extraction ........................................... 17
4.2.5 Fault Tolerance ............................................ 17
4.3 API Usage .................................................... 18
5 Case Study ..................................................... 20
5.1 Federated Averaging .......................................... 20
5.2 Asynchronous Averaging ....................................... 21
5.3 Customization ................................................ 22
6 Evaluation ..................................................... 25
6.1 Paper Result Reproduce ....................................... 25
6.1.1 Setup ...................................................... 25
6.1.2 Federated Averaging ........................................ 26
6.1.3 Distributed Selective SGD .................................. 28
6.2 Staleness .................................................... 30
6.3 Fault Tolerance .............................................. 32
6.4 Performance Comparison with PaddleFL ......................... 33
7 Conclusion ..................................................... 36
References ....................................................... 37

[1] Fate - Federated AI Ecosystem.
[2] Federated Deep Learning in PaddlePaddle.
[3] TensorFlow Federated: Machine Learning on Decentralized Data.
[4] Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Pa- tel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (New York, NY, USA, 2017), CCS ’17, Association for Computing Machinery, p. 1175‒ 1191.
[5] Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. Leaf: A benchmark for federated settings, 2018.
[6] Caldas, S., Konečny, J., McMahan, H. B., and Talwalkar, A. Expanding the reach of federated learning by reducing client resource requirements, 2018.
[7] Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. Revisiting dis- tributed synchronous sgd, 2016.
[8] Chen,X.,Chen,T.,Sun,H.,Wu,Z.S.,andHong,M.Distributedtrainingwith heterogeneous data: Bridging median- and mean-based algorithms, 2019.
[9] Chen,Y.,Sun,X.,andJin,Y.Communication-efficientfederateddeeplearning with asynchronous model update and temporally weighted aggregation, 2019.
[10] Geyer, R. C., Klein, T., and Nabi, M. Differentially private federated learning: A client level perspective, 2017.
[11] Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., and Xing, E. P. More effective distributed ml via a stale syn- chronous parallel parameter server. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1 (Red Hook, NY, USA, 2013), NIPS’13, Curran Associates Inc., p. 1223‒1231.
[12] Hu, C., Jiang, J., and Wang, Z. Decentralized federated learning: A segmented gossip approach, 2019.
[13] Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R. G. L., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konečný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Raykova, M., Qi, H., Ramage, D., Raskar, R., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. Advances and open problems in federated learning, 2019.
[14] Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication effi- ciency, 2016.
[15] Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks, 2018.
[16] Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. Deep gradient compres- sion: Reducing the communication bandwidth for distributed training, 2017.
[17] McMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data, 2016.
[18] Phong, L. T., Aono, Y., Hayashi, T., Wang, L., and Moriai, S. Privacy- preserving deep learning via additively homomorphic encryption. IEEE Trans- actions on Information Forensics and Security 13, 5 (2018), 1333–1345.
[19] Phong, L. T., Aono, Y., Hayashi, T., Wang, L., and Moriai, S. Privacy- preserving deep learning via additively homomorphic encryption. IEEE Trans- actions on Information Forensics and Security 13, 5 (2018), 1333–1345.
[20] Pillutla,K.,Kakade,S.M.,andHarchaoui,Z.Robustaggregationforfederated learning, 2019.
[21] Ryffel, T., Trask, A., Dahl, M., Wagner, B., Mancuso, J., Rueckert, D., and Passerat-Palmbach, J. A generic framework for privacy preserving deep learn- ing, 2018.
[22] Sattler, F., Wiedemann, S., Müller, K.-R., and Samek, W. Robust and communication-efficient federated learning from non-iid data, 2019.
[23] Shokri, R., and Shmatikov, V. Privacy-preserving deep learning. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton) (2015), pp. 909–910.
[24] Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A. Federated multi-task learning, 2017.
[25] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Jour- nal of Machine Learning Research 15, 56 (2014), 1929–1958.
[26] WANG,L.,WANG,W.,andLI,B.Cmfl:Mitigatingcommunicationoverhead for federated learning. pp. 954–964.
[27] Xie,C.,Koyejo,S.,andGupta,I.Asynchronousfederatedoptimization,2019. [28] Yuhong Wen, Wenqi Li, H. R., and Dogra, P. Federated Learning powered by
NVIDIA Clara.
[29] Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. Federated
learning with non-iid data, 2018.
[30] Zhu, L., Lu, Y., Lin, Y., and Han, S. Distributed training across the world, 2020.