低秩分解通常被用作深度卷積神經網絡的結構化模型壓縮方法，藉以探索並消除矩陣或張量內的線性相依性。然而，隨著壓縮比例超過一定閾值，模型的準確度會迅速下降。我們觀察到，在使用極低秩壓縮的卷積神經網絡中，只要加上少量稀疏元素，就能顯著地恢復模型的準確度。
基於這個前提，我們開發了一種新穎的方法，稱為LPSD（低秩加稀疏分解），它將卷積神經網路權重張量分解成低秩和稀疏分量的組合，在壓縮後能更好地保持準確性。對於預訓練模型，每一層的網絡結構被拆分為兩個分支：一低秩部分與稀疏部分。
基於LPSD，我們開發了兩種演算法：第一種稱為Sparsification（稀疏化）。Sparsification使用ALDS(自動分層分解秩選擇演算法)對原始模型進行低秩壓縮，以確定低秩部分，接著對原始模型與低秩部分的差進行全局稀疏化，以確定稀疏部分。
第二種演算法稱為Alternative LPSD（交替低秩加稀疏分解），它利用矩陣交替逼近算法同時確定低秩和稀疏部分。首先，它使用全局步驟來確定每一層的稀疏度，然後根據前面每層的選擇執行局部稀疏度選擇，獲得最終的稀疏度分布。
實驗結果顯示，在大多數情況下，我們的方法在更小的模型尺寸下實現了比最先進方法更好的準確性。同時，我們還提供了消融研究來評估不同超參數對模型的影響。

Low-rank decomposition that explores and eliminates the linear dependency within a matrix or a tensor is often used as a structured model compression method for deep convolutional neural networks. However, the model accuracy declines rapidly as the compression ratio decreases over a threshold. We have observed that with a small amount of sparse elements, the model accuracy can be recovered significantly for the CNN weight networks compressed with extremely low ranks.
Based on this premise, we developed a novel method, called LPSD (Low-rank Plus Sparse Decomposition), that decomposes a CNN weight tensor into a combination of low-rank and sparse components, which can better maintain the accuracy after the extremely low rank compression. For a pre-trained model, the network structure of each layer is split into two branches: one for low-rank part and one for sparse part. Based on LPSD, we have developed two algorithms: Sparsification and Alternative LPSD.
Sparsification employs ALDS to perform low-rank compression on the original model to determine the low-rank part. Additionally, it applies L1 global sparsification on the difference between the original model and the low-rank part to determine the sparse part.
Alternative LPSD utilizes the matrix alternating approximation algorithm to simultaneously determine the low-rank and sparse parts. It starts by using global selection to determine the sparsity for each layer, and then performs local selection of sparsity based on the previous per-layer selection to obtain the final sparsity distribution.
Experimental results demonstrate that in most scenarios, Our method achieves better accuracy with smaller model sizes compared to the state-of-the-art methods. Ablation studies are also provided to evaluate the impact of different hyper-parameters.

Abstract (Chinese) ---I
Abstract ---II
Contents ---III
List of Figures ---V
1 Introduction ---1
2 Related Works ---4
2.0.1 Low-rankness only ---4
2.0.2 Sparsity/Unstructured pruning only ---5
2.0.3 Combining low-rankness and sparsity ---5
3 LPSD Method ---6
3.0.1 Low-Rank Decomposition ---6
3.0.2 Sparsification ---9
3.0.3 Alternative LPSD ---10
4 Experiments ---15
4.0.1 Experimental Setting ---15
4.0.2 Comparison with other methods ---15
4.0.3 The experimental results of Sparsification and ALPSD under
different parameter settings ---17
4.0.4 Comparison between Sparsification and ALPSD ---19
5 Conclusion and Future Work ---22
Bibliography ---23

[1] Dimitris Bertsimas, Ryan Cory-Wright, and Nicholas A. G. Johnson. Sparse plus low rank matrix decomposition: A discrete optimization approach, 2023.
[2] Jian-Feng Cai, Jingyang Li, and Dong Xia. Generalized low-rank plus sparse tensor estimation by fast riemannian optimization, 2022.
[3] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation, 2014.
[4] Kailing Guo, Xiaona Xie, Xiangmin Xu, and Xiaofen Xing. Compressing by learning in a low-rank and sparse decomposition form. IEEE Access, 7:150823–150832, 2019.
[5] Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang,Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, and William J. Dally. Dsd: Dense-sparse-dense training for deep neural networks, 2017.
[6] Cole Hawkins, Haichuan Yang, Meng Li, Liangzhen Lai, and Vikas Chandra. Low-rank+sparse tensor compression for neural networks, 2021.
[7] Wenqi Huang, Ziwen Ke, Zhuo-Xu Cui, Jing Cheng, Zhilang Qiu, Sen Jia, Leslie Ying, Yanjie Zhu, and Dong Liang. Deep low-rank plus sparse network for dynamic mr imaging, 2021.
[8] Yerlan Idelbayev and Miguel A. Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8046–8056, 2020.
[9] Pavel Kaloshin. Convolutional neural networks compression with low rank and sparse tensor decompositions, 2020.
[10] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications, 2016.
[11] Lucas Liebenwein, Alaa Maalouf, Oren Gal, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. CoRR, abs/2107.11442, 2021.
[12] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback, 2020.
[13] Ricardo Otazo, Emmanuel Candès, and Daniel Sodickson. Low-rank plus sparse matrix decomposition for accelerated dynamic mri with separation of background and dynamic components. Magnetic Resonance in Medicine, 73, 04 2014.
[14] Miao Yin, Huy Phan, Xiao Zang, Siyu Liao, and Bo Yuan. Batude: Budgetaware neural network compression based on tucker decomposition. Proceedings of the AAAI Conference on Artificial Intelligence, 36:8874–8882, 06 2022.
[15] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 67–76, 2017.
[16] Xiao Zhang, Lingxiao Wang, and Quanquan Gu. A unified framework for low-rank plus sparse matrix recovery, 2018.