此論文考慮在異質和不可靠環境中使用分散式字典學習，這是對於[1]的分散式自適應字典學習的改進。我們提出了一種強健的分佈式字典學習算法，可以共同學習通用的全局字典，增強模型對不可靠客戶端的強健性，並構建局部自適應判別子字典。在不可靠的分佈式環境下，很難學習到強健的全局字典。所提出的具有本地自適應子字典的強健分佈式字典學習（RDDL-ASD）算法由三部分組成：學習全局字典以學習共同知識;將不可靠的特徵捕獲到本地結構干擾子字典;構建一個自適應本地具有鑑別力的子字典，藉由使用所提出的拆分方法來處理特定於任務的知識。我們的目標是讓本地不可靠的特徵學習到強健的全局字典。首先，在本地客戶端捕獲結構化干擾特徵，其次，度量客戶端的可靠性以重新加權上傳的字典以抑制剩餘的不可靠行為。實驗結果表明，我們提出的RDDL-ASD演算法可以在更加異質和不可靠的環境中取得更好的結果。

This work considers the use of distributed dictionary learning in heterogeneous and unreliable environments, which is an improvement of distributed adaptive dictionary learning for [1]. We propose a robust distributed dictionary learning algorithm that enables collaboratively learning of a common global dictionary, enhancing the model's robustness to unreliable clients, and constructing the local adaptive discriminative sub-dictionary. Under the unreliable distributed environment, it is hard to learn a robust global dictionary. The proposed robust distributed dictionary learning with adaptive sub-dictionaries (RDDL-ASD) algorithm consists of three parts: learning a consensus global dictionary to learn the common knowledge, capturing the unreliable features into a local structure disturbance sub-dictionary, and constructing an adaptive local discriminative class-specific sub-dictionary to deal with task-specific knowledge by using the proposed splitting procedure. We aim to let the unreliable feature in the local part learn the robust global dictionary. First, capture the structured disturbance feature at the local client without uploading, and second, measure the reliability of the client to re-weight the uploaded dictionary to suppress the remaining unreliable behavior. Experimental results demonstrated that our proposed RDDL-ASD algorithm can achieve a better result in a more heterogeneous and unreliable environment.

Abstract i
Contents ii
1 Introduction 1
2 Background and Related Works 5
2.1 Distributed Dictionary Learning . . . . . . . . . . . . . . .5
2.2 Multitask Dictionary Learning . . . . . . . . . . . . . . . .6
2.3 Discriminative Dictionary Learning . . . . . . . . . . . . . 7
2.4 Robust Dictionary Learning . . . . . . . . . . . . . . . . . 8
2.5 Dynamic Size of Dictionary Learning . . . . . . . . . . . . 10
3 System Model and Problem Formulation 12
3.1 System Model . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Review of DDL-LAD Algorithm . . . . . . . . . . . . . . . . 14
3.3 Problem Formulation. . . . . . . . . . . . . . . . . . . . .16
4 Robust Distributed Dictionary Learning with Local Adaptive Sub-Dictionaries 19
4.1 Client-Side and Server-Side Optimization . . . . . . . . . . 19
4.2 Distributed Dictionary Learning with Local Adaptive Discriminative Sub-Dictionaries 24
4.2.1 Expansion Procedure for Local Sub-Dictionary . . . . . . . 24
4.2.2 Pruning Procedure for Local Sub-Dictionary Elimination . . 26
4.3 Robust Global Dictionary with Hard and Soft decision . . . . 27
5 Experimental Result 30
5.1 Implementation: Setting, Datasets, Comparison Methods. . . . 30
5.2 Heterogeneous Task Result. . . . . . . . . . . . . . . . . . 32
5.3 Unreliable Environment Result. . . . . . . . . . . . . . . . 34
6 Conclusion and Future Work 40
Bibliography 41

[1] Y.-D. Huang, “Distributed dictionary learning over heterogeneous clients using local adaptive dictionaries,” Master’s thesis, National Tsing Hua University, 2021.
[2] P. Peng, Y. Tian, T. Xiang, Y. Wang, M. Pontil, and T. Huang, “Joint semantic and latent attribute modeling for cross-class transfer learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 40, no. 7, pp. 1625–1638, 2017.
[3] J. Liang, M. Zhang, X. Zeng, and G. Yu, “Distributed dictionary learning for sparse representation in sensor networks,” IEEE Transactions on Image Processing, vol. 23, no. 6, pp. 2528–2541, 2014.
[4] Z. Jiang, Z. Lin, and L. S. Davis, “Label consistent K-SVD: Learning a discriminative dictionary for recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 35, no. 11, pp. 2651–2664, 2013.
[5] A. Daneshmand, G. Scutari, and F. Facchinei, “Distributed
dictionary learning,” in Proceedings of Asilomar Conference on Signals, Systems and Computers (ACSSC), pp. 1001–1005, 2016.
[6] H. Raja and W. U. Bajwa, “Cloud K-SVD: A collaborative dictionary learning algorithm for big, distributed data,” IEEE Transactions on Signal Processing, vol. 64, no. 1, pp. 173–188, 2015.
[7] J. Chen, Z. J. Towfic, and A. H. Sayed, “Dictionary learning over distributed models,” IEEE Transactions on Signal Processing, vol. 63, no. 4, pp. 1001–1016, 2014.
[8] A. Koppel, G. Warnell, E. Stump, and A. Ribeiro, “D4l: Decentralized dynamic discriminative dictionary learning,” IEEE Transactions on Signal and Information Processing over Networks, vol. 3, no. 4, pp. 728–743, 2017.
[9] B.-S. Shin, M. Yukawa, R. L. Cavalcante, and A. Dekorsy, “A hybrid dictionary approach for distributed kernel adaptive filtering in diffusion networks,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3414–3418, 2018.
[10] A. Kumar and H. Daume III, “Learning task grouping and overlap in multi-task learning,” arXiv preprint arXiv:1206.6417, 2012.
[11] A. Maurer, M. Pontil, and B. Romera-Paredes, “Sparse coding for multitask and transfer learning,” in Proceedings of International Conference on Machine Learning (ICML), pp. 343–351, 2013.
[12] B.Wang and J. Pineau, “Generalized dictionary for multitask learning with boosting.,” in Proceedings of International Joint Conferences on Artificial Intelligence Organization (IJCAI), pp. 2097–2103, 2016.
[13] J. Wang, M. Kolar, and N. Srerbo, “Distributed multi-task learning,” in Artificial intelligence and statistics, pp. 751–760, Proceedings of Machine Learning Research, 2016.
[14] A. Fawzi, M. Sinn, and P. Frossard, “Multitask additive models with shared transfer functions based on dictionary learning,” IEEE Transactions on Signal Processing, vol. 65, no. 5, pp. 1352–1365, 2017.
[15] S. Kong and D. Wang, “A dictionary learning approach for classification: Separating the particularity and the commonality,” in European conference on computer vision, pp. 186–199, Springer, 2012.
[16] Z. Li, Z. Zhang, J. Qin, Z. Zhang, and L. Shao, “Discriminative fisher embedding dictionary
learning algorithm for object recognition,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 3, pp. 786–800, 2020.
[17] Y. Sun, Z. Zhang, W. Jiang, Z. Zhang, L. Zhang, S. Yan, and M. Wang, “Discriminative local sparse representation by robust adaptive dictionary pair learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 10, pp. 4303–4317, 2020.
[18] S. Mukherjee, R. Basu, and C. S. Seelamantula, “1-k-svd: A robust dictionary learning algorithm with simultaneous update,” Signal Processing, vol. 123, pp. 42–52, 2016.
[19] G. Lin, M. Yang, J. Yang, L. Shen, and W. Xie, “Robust, discriminative and comprehensive dictionary learning for face recognition,” Pattern Recognition, vol. 81, pp. 341–356, 2018.
[20] Z. Chen and Y.Wu, “Robust dictionary learning by error source decomposition,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2216–2223, 2013.
[21] P. Zhou, C. Fang, Z. Lin, C. Zhang, and E. Y. Chang, “Dictionary learning with structured noise,” Neurocomputing, vol. 273, pp. 414–423, 2018.
[22] R. Mazhar and P. D. Gader, “EK-SVD: Optimized dictionary design for sparse representations,” in Proceedings of International Conference on Pattern Recognition (ICPR), pp. 1–4, 2008.
[23] J. Feng, L. Song, X. Yang, and W. Zhang, “Sub clustering K-SVD: size variable dictionary learning for sparse representations,” in IEEE International Conference on Image Processing (ICIP), pp. 2149–2152, 2009.
[24] C. Rusu and B. Dumitrescu, “Stagewise K-SVD to design efficient dictionaries for sparse representations,” IEEE Signal Processing Letters, vol. 19, no. 10, pp. 631–634, 2012.
[25] M. Marsousi, K. Abhari, P. Babyn, and J. Alirezaie, “An adaptive approach to learn overcomplete dictionaries with efficient numbers of elements,” IEEE Transactions on Signal Processing, vol. 62, no. 12, pp. 3272–3283, 2014.
[26] J. Feng, L. Song, X. Yang, and W. Zhang, “Learning dictionary via subspace segmentation for sparse representation,” in 2011 18th IEEE International Conference on Image Processing, pp. 1245–1248, IEEE, 2011.
[27] B. Dumitrescu and P. Irofti, “Low dimensional subspace finding via size-reducing dictionary learning,” in 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6, 2016.
[28] R. E. Rolon, L. E. Di Persia, R. D. Spies, and H. L. Rufiner, “A multi-class structured dictionary learning method using discriminant atom selection,” Pattern Analysis and Applications, vol. 24, no. 2, pp. 685–700, 2021.
[29] W.Wang, Y. Yan, S.Winkler, and N. Sebe, “Category specific dictionary learning for attribute specific feature selection,” IEEE Transactions on Image Processing, vol. 25, no. 3, pp. 1465–1478, 2016.
[30] L. Yang, J. Dong, J. Zhang, X. Luo, and J. Guan, “Distributed analysis dictionary learning based on consensus constraints,” in 2021 IEEE 6th International Conference on Signal and Image Processing (ICSIP), pp. 779–783, 2021.