經由深度學習，有非常多領域已經達到了非常好的成果，像是分類問題，物件偵測甚至是物件認知，都在深度學習的使用範疇中。然而，我們知道深度學習有個缺點，就是他的參數量非常多，而這能夠幫助他學得非常好。隨著時代進步，如果我們想要將這項技術應用於攜帶型裝置或是小型裝置，他的參數量實在是無法負荷。於是，降低他的參數量以及使得他運算的更快將是我們的目標。
在這篇論文中，我們將提出一個能適用於任何模型的壓縮方法。其他的壓縮方法，像是去除模型中的濾波器，雖然能夠達到良好的壓縮效率，但是這只能用於那些擁有著冗餘濾波器的模型；另一種為量化壓縮，雖然能夠強勢地減少所需的參數量，但是也需要額外的數學程式庫來輔助計算。這篇論文所推薦的方法是利用一個原本就做得好的模型去訓練一個較小較差的模型，藉由引導資訊使得小模型成長到跟大模型一樣好。為此，我們需要找到關鍵的層來傳遞資訊。我們實驗了二種模型，一為假指紋偵測，用來避免個人資料遭竊的系統，另一個為影像生成模型，我們的方法在一定程度的犧牲下適用於任何模型。

In recent years, deep learning has been shown to achieve phenomenal performance in various fields of research including objection detection as well as image-based classification. However, one of the drawbacks of deep learning algorithms is that a deeper network may result in better performance but comes with a price of a larger number of parameters. It means it requires more hardware resources and longer running time. This causes an issue if deep learning is to be applied on portable devices with real-time processing requirement. One of the countermeasures is to design a light-weight deep learning model for the application.
In this thesis, we propose a deep learning model compression method that can be applied to every model theoretically. Other compression methods like filter trimming may yield good performance only if the model itself contains redundant layers, and quantization method may provide great compression rate but its performance deteriorates more than our method and it needs additional math libraries. Our method compresses a deep learning model by consulting a better model, known as the teacher model, which is larger than the target model, the student model. We coin this technique as information guide. To train the student model, we need to find the key layers to extract the information that the teacher is supposed to transfer to the student. We test three models with different datasets. One is fingerprint liveness detection, a system that protects fingerprint recognition systems from spoofing attack and the other is an image generator model. Our experiments show that the proposed method can be conveniently applied to the tested models with negligible performance drop.

摘要 I
ABSTRACT II
CHAPTER 1. INTRODUCTION 1
1.1 MOTIVATION 1
1.2 PROBLEM DESCRIPTION 2
1.3 MAIN CONTRIBUTIONS 3
1.4 THESIS ORGANIZATION 4
CHAPTER 2. RELATED WORKS 5
2.1 FILTER-PRUNING NETWORKS 5
2.2 WEIGHT-PRUNING AND ENCODING NETWORKS 6
2.3 DISTILLATION NETWORKS 7
2.4 INFORMATION GUIDE (IG) 8
CHAPTER 3. PROPOSED METHOD 10
3.1 KEY LAYER 12
3.2 TRAINING STRATEGY 12
CHAPTER 4. EXPERIMENTAL EVALUATION 15
4.1 TRAINING ENVIRONMENT 16
4.2 FINGERPRINT LIVENESS DETECTION 16
4.2.1 Dataset 17
4.2.2 Training Details 18
4.2.3 Experiment Result 19
4.3 VDSR 26
4.3.1 Dataset 26
4.3.2 Training Details 26
4.3.3 Experiment Result 27
CHAPTER 5. CONCLUSION 39
REFERENCES 40

[1] Li, Hao, et al. "Pruning Filters for Efficient ConvNets." arXiv preprint arXiv:1608.08710 (2016).
[2] Molchanov, Pavlo, et al. "Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning." arXiv preprint arXiv:1611.06440 (2016).
[3] Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al. Predicting parameters in deep learning. In NIPS, 2013.
[4] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).
[5] Rastegari, Mohammad, et al. "Xnor-net: Imagenet classification using binary convolutional neural networks." European Conference on Computer Vision. Springer International Publishing, 2016.
[6] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
[7] Romero, Adriana, et al. "Fitnets: Hints for thin deep nets." arXiv preprint arXiv:1412.6550 (2014).
[8] Bevilacqua, Marco, et al. "Low-complexity single-image super-resolution based on nonnegative neighbor embedding." (2012): 135-1.
[9] Zeyde, Roman, Michael Elad, and Matan Protter. "On single image scale-up using sparse-representations." International conference on curves and surfaces. Springer Berlin Heidelberg, 2010.
[10] Huang, Jia-Bin, Abhishek Singh, and Narendra Ahuja. "Single image super-resolution from transformed self-exemplars." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.
[11] Ghiani, Luca, et al. "Livdet 2013 fingerprint liveness detection competition 2013." Biometrics (ICB), 2013 International Conference on. IEEE, 2013.
[12] Yambay, David, et al. "LivDet 2011—Fingerprint liveness detection competition 2011." Biometrics (ICB), 2012 5th IAPR International Conference on. IEEE, 2012.
[13] Marcialis, Gian, et al. "First international fingerprint liveness detection competition—livdet 2009." Image Analysis and Processing–ICIAP 2009 (2009): 12-23.
[14] Goyal, Priya, et al. "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." arXiv preprint arXiv:1706.02677 (2017).
[15] Ullrich, Karen, Edward Meeds, and Max Welling. "Soft weight-sharing for neural network compression." arXiv preprint arXiv:1702.04008 (2017).
[16] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
[17] Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
[18] Nogueira, Rodrigo Frassetto, Roberto de Alencar Lotufo, and Rubens Campos Machado. "Fingerprint liveness detection using convolutional neural networks." IEEE Transactions on Information Forensics and Security 11.6 (2016): 1206-1213.
[19] Krizhevsky, Alex, and Geoffrey Hinton. "Learning multiple layers of features from tiny images." (2009).