此碩論探討分散式的矩陣分解問題，有著資料被存放在各個不同的地方，且資料本身並不會傳送到各個地方。大部分的研究思考透過平行計算的方式來加速處理矩陣分解問題，但卻忽略了資料隱私性和傳輸資料量的問題。我們提出一個有傳輸效率的交替方向乘子法(ADMM)搭配交替最佳化(alternating optimization)，對使用者潛在特徵矩陣(users’latent feature matrices)和物品潛在特徵矩陣(items’latent feature matrices)輪流地更新。關鍵的想法是利用
有限元量化的方式來減少每單位被傳輸訊息的位元數和通訊審核(communication censoring)技術來限制是否此訊息有足夠多資訊，使得需要被傳輸出去。收斂的結果和錯誤上界因為量化和通訊審核在此論文也有被提供。除此之外，我們也提出一個有隱私保存效果的方法，想法是透過晶格點(lattices)被分成好幾組副晶格點(sublattices)，且只有此組被傳輸副晶格點的索引(index)被傳送，最後使用被傳輸地方的資料來重建被傳輸的訊息。實驗在兩組真實的資料:MovieLens100K和Yahoo Movie R4，結果顯示我們提出的方法可以在不犧牲太多準確率的情況下節省更多的溝通量。

This work examines a distributed matrix factorization (MF) problem where data is stored locally at different sites and is not exchanged explicitly with other sites or the data center. Most existing works on distributed MF focus on parallelizing the computation and, thus, do not explicitly address concerns regarding data privacy and communication costs. This work proposes a communication-efficient alternating optimization alternating direction method of multipliers (ADMM) algorithm, in which the users' and the items' latent feature matrices are optimized in turn using a reduced-messaging ADMM algorithm. The key idea is to utilize finite-bit quantization to reduce the number of bits exchanged in each message, and communication-censoring to restrict the local nodes' transmissions only to cases where their updates are sufficiently informative. The convergence results and error bounds due to quantization and censoring are both provided. Besides, the work also proposes a privacy-preserving quantization scheme where the lattice of the original quantizer is split into multiple sublattices and only the indices of the message's sublattices are exchanged in each iteration. The local data is then utilized as side information to reconstruct the original message. Experiments on two real-world datasets, namely MovieLens100K and Yahoo Movie R4, show a significant reduction in the communication overhead while achieving high prediction accuracy.

Abstract i
Contents ii
1 Introduction 1
2 Related Works 4
2.1 Related Works on Parallel Algorithms for Matrix Factorization . . . . . . . . 4
2.2 Related Works on Quantized ADMM . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Related Works on ADMM with Communication Censoring . . . . . . . . . . 6
2.4 Related Works on ADMM with Privacy . . . . . . . . . . . . . . . . . . . . . 7
3 System Model 8
3.1 System Model With Centralized Version . . . . . . . . . . . . . . . . . . . . 9
3.2 System Model With Distributed Version . . . . . . . . . . . . . . . . . . . . 9
4 AO-ADMM for Distributed Matrix Factorization 11
4.1 Review of the ADMM Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 AO-ADMM with Master-slave Structure . . . . . . . . . . . . . . . . . . . . 12
5 Quantized Censoring ADMM for Distributed Matrix Factorization 16
5.1 Quantization and Censoring Operations . . . . . . . . . . . . . . . . . . . . . 16
5.2 The Convergence Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 Consensus Error Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3.1 The Error Bound of Finite-Bit Quantizer . . . . . . . . . . . . . . . . 20
5.3.2 The Error Bound of In_nite-Bit Quantizer . . . . . . . . . . . . . . . 20
6 Quantized ADMM with Privacy-Preserving 22
7 Experimental Result 25
7.1 Experiments on the MovieLens100K Dataset . . . . . . . . . . . . . . . . . . 26
7.2 Experiments on the Yahoo Movie R4 Dataset . . . . . . . . . . . . . . . . . 31
8 Conclusion 35
Appendices 36
A The Derivation of QC-ADMM to Distributed Form 37
B Proof of Theorem 1 42
C Proof of Theorem 2 48
D Proof of Theorem 3 51
Bibliography 55

[1] Y. Koren, R. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” Computer, vol. 42, pp. 30~37, August. 2009.
[2] A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky, “Tensor decompositions for learning latent variable models,” Journal of Machine Learning Research, vol. 15, pp. 2773-2832, 2014.
[3] D. Nion, K. N. Mokios, N. D. Sidiropoulos, and A. Potamianos, “Batch and adaptive PARAFAC-based blind separation of convolutive speech mixtures,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, pp. 1193-1207, August. 2010.
[4] Y. Zhou, D. Wilkinson, R. Schreiber, and R. Pan, “Large-scale parallel collaborative filtering for the netflix prize,” in Proceedings of Algorithmic Aspects in Information and Management (R. Fleischer and J. Xu, eds.), (Berlin, Heidelberg), pp. 337-348, Springer,2008.
[5] R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis, “Large-scale matrix factorization with distributed stochastic gradient descent,” in Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 69-77,2011.
[6] H.-F. Yu, C.-J. Hsieh, S. Si, and I. S. Dhillon, “Parallel matrix factorization for recommender systems," Knowledge and Information Systems, vol. 41, pp. 793-819, December.2014.
[7] Y. Qian, C. Tan, N. Mamoulis, and D. W. Cheung, “Dsanls: Accelerating distributed nonnegative matrix factorization via sketching,” in Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM), pp. 450-458, 2018.
[8] A. P. Liavas and N. D. Sidiropoulos, “Parallel algorithms for constrained tensor factorization via alternating direction method of multipliers,” IEEE Transactions on Signal Processing, vol. 63, pp. 5450-5463, October. 2015.
[9] K. Huang, N. D. Sidiropoulos, and A. P. Liavas, “A flexible and efficient algorithmic
framework for constrained matrix and tensor factorization,” IEEE Transactions on Signal Processing, vol. 64, pp. 5052{5065, October. 2016.
[10] S. Zhu, M. Hong, and B. Chen, “Quantized consensus admm for multi-agent distributed optimization," in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pp. 4134-4138, March 2016.
[11] S. Zhu and B. Chen, “Distributed average consensus with bounded quantization,” in Proceedings of the IEEE 17th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), pp. 1-6, July 2016.
[12] J. Matamoros, S. M. Fosson, E. Magli, and C. Ant_on-Haro, “Distributed admm for in-network reconstruction of sparse signals with innovations,” IEEE Transactions on Signal Processing, vol. 1, no. 4, pp. 225-234, 2015.
[13] Y. Liu, W. Xu, G. Wu, Z. Tian, and Q. Ling, “Communication-censored admm for decentralized consensus optimization,” IEEE Transactions on Signal Processing, vol. 67, pp. 2565-2579, May 2019.
[14] W. Li, T. Chen, L. Li, and Q. Ling, “Communication-censored distributed stochastic gradient descent,” arXiv preprint arXiv:1909.03631, 2019.
[15] P. Xu, Z. Tian, Z. Zhang, and Y. Wang, “Coke: Communication-censored kernel learning via random features,” in 2019 IEEE Data Science Workshop (DSW), pp. 32-36, June 2019.
[16] J. Hua, C. Xia, and S. Zhong, “Differentially private matrix factorization," in Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pp. 1763-1770, 2015.
[17] F. Zhang, V. Lee, and K.-K. Choo, “Jo-dpmf: Differentially private matrix factorization learning through joint optimization," Information Sciences, vol. 467, 07 2018.
[18] E. Xue, R. Guo, F. Zhang, L. Wang, X. Zhang, and G. Qu, \Distributed differentially private matrix factorization based on admm," in 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International
Conference on Smart City; IEEE 5th International Conference on Data Science and
Systems (HPCC/SmartCity/DSS), pp. 2502{2507, Aug 2019.
[19] Z. Huang, R. Hu, Y. Guo, E. Chan-Tin, and Y. Gong, “Dp-admm: Admm-based distributed learning with differential privacy," IEEE Transactions on Information Forensics and Security, vol. 15, pp. 1002-1012, 2020.
[20] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning, vol. 3, no. 1, pp. 1{122, 2011.
[21] W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin, “On the linear convergence of the
admm in decentralized consensus optimization,” IEEE Transactions on Signal Processing, vol. 62, pp. 1750{1761, April 2014.
[22] S.Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan,”Adaptive federated learning in resource constrained edge computing systems,” IEEE Journal on Selected Areas in Communications, vol. 37, pp. 1205-1221, June 2019.
[23] F. M. Harper and J. A. Konstan, “The movielens datasets: History and context,” ACM Trans. Interact. Intell. Syst., vol. 5, no. 4, pp. 19:1-19:19, 2015.
[24] “Yahoo!: Webscope movie data set (version 1.0), http://research.yahoo.com/,”
[25] F. Li, B. Wu, L. Xu, C. Shi, and J. Shi, “A fast distributed stochastic gradient descent algorithm for matrix factorization,” in BigMine, vol. 36 of JMLR Workshop and Conference Proceedings, pp. 77-87, JMLR.org, 2014.