近年來，將深度學習模型部屬到終端產品是十分熱門的議題，在實際場景中會考慮到模型運行時之計算量、運算時間、消耗功率，因此模型壓縮方法逐漸受到學界重視。此外，現實情境中之訓練資料集可能會面臨傳輸量限制、法律、隱私而無法取得，這將使一般基於資料驅動所設計(Data-driven)的模型壓縮方法難以直接應用。
Zhang 等人提出了一套無資料超解析度模型壓縮的方案，然而在實作知識蒸餾壓縮學生模型時，他們使用了多個圖像生成器產生不同解析度尺度的生成圖像以當作訓練學生模型的圖像來源，然而，我們認為Zhang 所使用的生成器可能存在著冗餘項或者說生成器之間存在著相似的信息，因此我們提出了複用生成器的架構，降低知識蒸餾壓縮學生模型時所需要的生成器數量。經實驗顯示，本文提出的架構能以較少數量的生成器達到與Zhang 的方法(未複用生成器架構)相當的超解析度性能，且本架構亦能緩和在終端訓練的情境下，儲存資源有限的瓶頸。

Recently, deploying deep learning model to edge products has become a very popular topic. In real-world scenarios, the computation amount, operation time, and power consumption of the model will be considered. Therefore, the model compression method has gradually attracted the attention of the academic. In addition, training data sets in real-world situations may not be available due to limitation of transmission throughput, legal, or privacy, which will make it difficult to directly apply the general data-driven model compression methods.
Zhang et al. proposed a data-free super-resolution model compression scheme. However, when implementing knowledge distillation to compress student model, they used multiple image generators to produce generated images of different resolution
scales as image sources for training the student model. However, we believe that the generators used by Zhang may have redundancy, in other words, there is similar information between generators, so we propose the architectures to reuse generators, which can reduce the number of generators required for knowledge distillation to compress student models. Experiments show that the architecture proposed in this paper can achieve super-resolution performance comparable to Zhang's method (non-reused generator architecture) with fewer generators. In addition, the proposed method can also save the precious storage resources when deploying the model to the edge devices for training.

摘要 i
Abstract ii
致謝 iii
目錄 iv
圖索引 vi
表索引 vii
第一章 緒論 1
第二章 相關研究 2
2.1 深度學習模型壓縮 2
2.1.1 模型剪枝 2
2.1.2 模型量化 3
2.2 知識蒸餾 4
2.2.1 無資料知識蒸餾 5
2.2.2 超解析度知識蒸餾 6
第三章 系統架構 7
3.1 概述 7
3.2 複用生成器架構 7
3.2.1 未複用生成器架構StudentG2346 8
3.2.2 複用一個生成器架構StudentG234 9
3.2.3 複用兩個生成器架構StudentG24 10
3.2.4 複用三個生成器架構StudentG2 11
3.2.5 生成器更新策略 12
3.3 先前知識蒸餾架構 14
3.4 超解析度神經網路模型 15
第四章 實驗與結果 17
4.1 資料集 17
4.1.1 訓練資料集 17
4.1.2 驗證資料集 17
4.1.3 測試資料集 18
4.2 實驗方法與參數設定 18
4.2.1 預訓練老師模型 18
4.2.2 訓練學生網路與生成器 21
4.3 實驗結果與評估指標 22
4.3.1 評估指標:PSNR 22
4.3.2 實驗結果 23
第五章 結論 28
參考文獻 29

[1] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015.
[2] D. Chen et al., "Cross-layer distillation with semantic calibration," in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, no. 8, pp. 7028-7036.
[3] S. Zagoruyko and N. Komodakis, "Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer," arXiv preprint arXiv:1612.03928, 2016.
[4] Z. He, T. Dai, J. Lu, Y. Jiang, and S.-T. Xia, "Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution," in 2020 IEEE International Conference on Image Processing (ICIP), 2020: IEEE, pp. 518-522.
[5] R. G. Lopes, S. Fenu, and T. Starner, "Data-free knowledge distillation for deep neural networks," arXiv preprint arXiv:1710.07535, 2017.
[6] H. Chen et al., "Data-free learning of student networks," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3514-3522.
[7] G. K. Nayak, K. R. Mopuri, V. Shaj, V. B. Radhakrishnan, and A. Chakraborty, "Zero-shot knowledge distillation in deep networks," in International Conference on Machine Learning, 2019: PMLR, pp. 4743-4751.
[8] G. Fang, J. Song, C. Shen, X. Wang, D. Chen, and M. Song, "Data-free adversarial distillation," arXiv preprint arXiv:1912.11006, 2019.
[9] Y. Zhang, H. Chen, X. Chen, Y. Deng, C. Xu, and Y. Wang, "Data-free knowledge distillation for image super-resolution," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7852-7861.
[10] J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1646-1654.
[11] A. Vaswani et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[12] T. Brown et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[13] S. Han, J. Pool, J. Tran, and W. Dally, "Learning both weights and connections for efficient neural network," Advances in neural information processing systems, vol. 28, 2015.
[14] B. Jacob et al., "Quantization and training of neural networks for efficient integer-arithmetic-only inference," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2704-2713.
[15] J. Ye, Y. Ji, X. Wang, X. Gao, and M. Song, "Data-free knowledge amalgamation via group-stack dual-gan," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12516-12525.
[16] H. Yin et al., "Dreaming to distill: Data-free knowledge transfer via deepinversion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8715-8724.
[17] Q. Gao, Y. Zhao, G. Li, and T. Tong, "Image super-resolution using knowledge distillation," in Asian Conference on Computer Vision, 2018: Springer, pp. 527-541.
[18] W. Lee, J. Lee, D. Kim, and B. Ham, "Learning with privileged information for efficient image super-resolution," in European Conference on Computer Vision, 2020: Springer, pp. 465-482.
[19] Y. Wang et al., "Towards compact single image super-resolution via contrastive self-distillation," arXiv preprint arXiv:2105.11683, 2021.
[20] Z. Xiao, X. Fu, J. Huang, Z. Cheng, and Z. Xiong, "Space-time distillation for video super-resolution," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2113-2122.
[21] C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295-307, 2015.
[22] S. Schulter, C. Leistner, and H. Bischof, "Fast and accurate image upscaling with super-resolution forests," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3791-3799.
[23] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, "Low-complexity single-image super-resolution based on nonnegative neighbor embedding," 2012.
[24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics," in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, 2001, vol. 2: IEEE, pp. 416-423.
[25] J. Yang, J. Wright, T. S. Huang, and Y. Ma, "Image super-resolution via sparse representation," IEEE transactions on image processing, vol. 19, no. 11, pp. 2861-2873, 2010.
[26] "PyTorch VDSR." [Online]. Available: https://github.com/twtygqyy/pytorch-vdsr.
[27] "PyTorch implementation of DFSR." [Online]. Available: https://github.com/huawei-noah/Efficient-Computing/tree/master/Data-Efficient-Model-Compression/DFSR.
[28] "bicubic.py." [Online]. Available: https://github.com/tonyzzzt/bicubic-interpolation-pytorch-version-the-same-results-with-matlab-imresize/blob/master/bicubic.py