本論文提出一套適用於卷積式類神經網路的壓縮方法。此壓縮法將對類神經網路進行逐層壓縮，並使用到「壓縮層」幫助我們過濾掉冗贅的變數。具體來說，我們會在原有的類神經網路中安插許多的壓縮層。每一個壓縮層的卷積核經過二元化之後恰好就是該卷積層對應的壓縮遮罩，演算法會根據壓縮遮罩來卷積層中刪除不重要地變數。除此之外，為了讓演算法能自動並有效率地進行壓縮，本文提出了一套成本考量機制以便學習適合每個卷積層的壓縮率，同時避免過度壓縮而犧牲了辨識準確率。最後，我們在物件辨識的壓縮成果可以達到現在最佳方法的水準。

This thesis describes a channel selection approach for compressing deep neural networks. Specifically, we propose a new type of generic network layer, called pruning layer, to seamlessly augment a given pre-trained model for compression. Each pruning layer, comprising 1x1 depth-wise kernels, is represented with a dual format: one is real-valued and the other is binary. The former enables a two-phase optimization process of network pruning to operate with an end-to-end differentiable network, and the latter yields the mask information for channel selection. Our method progressively performs the pruning task layer-wise, and achieves channel selection according to a sparsity criterion to favor pruning more channels. To make our method automatic and effective, we design a cost-aware mechanism to determine the proper sparsity for each convolution layer and avoid excessive pruning of weight parameters. Our results for compressing several benchmark deep networks on image classification are comparable to those by state-of-the-art.

摘要 6
Abstract 7
1 Introduction 8
2 Related work 11
3 Proposed method 14
3.1 Progressive network pruning ............ 14
3.2 Channel sparse selection ............... 16
3.3 Cost-aware mechanism ................... 19
4 Experiment 21
4.1 Experimental settings ............. .... 21
4.2 More on C2S2 ........................... 22
4.3 VGG-Net on CIFAR-10 .................... 24
4.4 ResNet-20 on CIFAR-10 .................. 25
4.5 ResNet-18 on ImageNet .................. 26
4.6 FCN on PASCAL VOC 2011 ................. 28
5 Conclusion 30
A Pruning results of VGG-Net on CIFAR-10 31
B Pruning results of ResNet-20 on CIFAR-10 33
C Pruning results of ResNet-18 on ImageNet 34

[1] A. Aghasi, A. Abdi, N. Nguyen, and J. Romberg. Net-trim: Convex pruning of deep neural networks
with performance guarantee. In Advances in Neural Information Processing Systems 30, pages 3180–
3189, 2017.
[2] J. M. Alvarez and M. Salzmann. Learning the number of neurons in deep networks. In Advances in
Neural Information Processing Systems 29, pages 2262–2270, 2016.
[3] H. Bagherinezhad, M. Rastegari, and A. Farhadi. LCNN: lookup-based convolutional neural network.
CoRR, abs/1611.06473, 2016.
[4] Z. Cai, X. He, J. Sun, and N. Vasconcelos. Deep learning with low precision by half-wave gaussian
quantization. CoRR, abs/1702.00953, 2017.
[5] G. Chechik, I. Meilijson, and E. Ruppin. Synaptic pruning in development: A computational account.
Neural Computation, 10(7):1759–1777, 1998.
[6] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li. Imagenet: A large-scale hierarchical image
database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255, 2009.
[7] B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon: Extensions and performance compari-
son. In Advances in Neural Information Processing Systems 6, pages 263–270, 1993.
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-
30, 2016, pages 770–778, 2016.
[9] Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. CoRR,
abs/1707.06168, 2017.
[10] G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. CoRR,
abs/1503.02531, 2015.
[11] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and
H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR,
abs/1704.04861, 2017.
[12] A. Krizhevsky. Learning multiple layers of features from tiny images.
[13] Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information
Processing Systems 2, pages 598–605, 1989.
[14] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. CoRR,
abs/1608.08710, 2016.
[15] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks
through network slimming. In IEEE International Conference on Computer Vision, ICCV 2017, Venice,
Italy, October 22-29, 2017, pages 2755–2763, 2017.
[16] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation.
abs/1411.4038, 2014.
[17] J. Luo, J. Wu, and W. Lin. ThiNet: A filter level pruning method for deep neural network compression.
In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017,
pages 5068–5076, 2017.
[18] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
CoRR, abs/1409.1556, 2014.
[19] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks.
CoRR, abs/1608.03665, 2016.
[20] A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen. Incremental network quantization: Towards lossless cnns
with low-precision weights. CoRR, abs/1702.03044, 2017.
[21] C. Zhu, S. Han, H. Mao, and W. J. Dally. Trained ternary quantization. CoRR, abs/1612.01064, 2016.