伴隨著人工智慧的快速發展，目前在工業製造領域已有許多企業計劃導入深度學習及物聯網等技術以提升效率、降低生產/服務所需的整體成本，例如：在半導體製造領域中設備導入感測器，並以即時資料來進行機台安全監控。其中，卷積神經網路(Convolution Neural Network, CNN)為機器視覺領域的一種常用技術，此方法透過前饋神經網路訓練學習圖像中的特徵資訊，檢測精度遠超過人工提取特徵的算法。但隨著準確精度要求不斷提高，訓練所得到的參數資訊會隨著CNN的深度加深而呈指數增加，使得將數據處理從雲端轉移到邊緣端網路節點上時，難以在運算資源有限的環境中部署深度學習模型。因此在儲存空間、計算資源及能源上有所限制的邊緣端裝置，將深度學習模型進行輕量化已然是理論轉向工程的重要過程。
LeCun於1989年中提出可以將神經網路中不重要的參數剔除來達到模型壓縮的效果，模型剪枝則是其中一種減少損失精度的關鍵壓縮技術。但目前大部分剪枝方法類似於在參數空間內的暴力搜尋法，僅能找到極少的有效組合；且以權重為修剪單位的非結構化剪枝，多數都無法達到壓縮及加速的效果。因此，本研究提出一個透過簡化群體演算法來對CNN進行結構化剪枝的方法SSO-Prune。首先，透過擾動分析找出網路的敏感度資訊，並將此資訊引入簡化群體演算法，然後使用簡單的權重累加評量方法選出候選的剪枝目標，最後再進行迭代搜尋並進行優化來找到最佳的壓縮結構。在使用公開資料集驗證後可發現，SSO-Prune結合啟發式和結構化剪枝的方法能夠有效率的在搜尋空間中找到更好的剪枝組合，並且在準確率不受影響甚至提升的情況上，大幅減少模型中的參數量，從而提升模型的每秒浮點運算次數

With the rapid development of artificial intelligence, many companies in the industrial manufacturing sector have planned to adopt technologies such as deep learning and IoT to enhance efficiency and reduce overall costs. Among various techniques, Convolutional Neural Network (CNN) is a commonly used method in the field of machine vision. However, as the demand for higher accuracy continues to rise, the parameter information obtained during training exponentially increases with the depth of the CNN. To address these limitations, lightweighting deep learning models has become an essential process.
In 1989, LeCun proposed a model compression technique that eliminates unimportant parameters in neural networks, known as model pruning. However, most existing pruning methods resemble brute-force search within the parameter space, resulting in the discovery of only a few effective combinations. Therefore, this research presents SSO-Prune, a structured pruning method for CNNs based on simplified swarm optimization. The proposed method incorporates sensitivity information obtained through perturbation analysis into the simplified swarm optimization algorithm. It utilizes a simple weight accumulation evaluation method to select candidate pruning targets, and iteratively searches and optimizes for the best compression structure. Experimental validation using publicly available datasets demonstrates that SSO-Prune, combining heuristic search and structured pruning, efficiently discovers better pruning combinations within the search space. Moreover, it significantly reduces the number of parameters in the model while maintaining or even improving accuracy, thereby enhancing the model's floating-point operations per second

摘要 i
Abstract ii
目錄 iii
圖目錄 v
表目錄 vi
第一章 緒論 1
1.1 研究背景、動機 1
1.2 研究目的 3
1.3 研究架構 5
第二章 文獻回顧 7
2.1 卷積神經網路 7
2.2 模型剪枝 14
2.3 混合精度訓練 18
2.4 簡化群體演算法 19
第三章 研究方法 21
3.1 粒子編碼方式 21
3.2 適應度函數 23
3.3 SSO-Prune 25
3.4 Fine-Tuning策略 28
第四章 實驗結果與分析 30
4.1 資料集及實驗環境說明 30
4.2 參數設定實驗設計 31
第五章 結論與未來研究方向 41
5.1 結論 41
5.2 未來研究規劃 41
參考文獻 42

[1] G. E. Moore, "Cramming more components onto integrated circuits," ed: McGraw-Hill New York, 1965.
[2] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," nature, vol. 323, no. 6088, pp. 533-536, 1986.
[3] Y. LeCun et al., "Backpropagation applied to handwritten zip code recognition," Neural computation, vol. 1, no. 4, pp. 541-551, 1989.
[4] W. Zaremba, I. Sutskever, and O. Vinyals, "Recurrent neural network regularization," arXiv preprint arXiv:1409.2329, 2014.
[5] I. Goodfellow et al., "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp. 139-144, 2020.
[6] K. Chen and X. Huang, "Feature extraction method of 3D art creation based on deep learning," Soft Computing, vol. 24, no. 11, pp. 8149-8161, 2020.
[7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834-848, 2017.
[8] M. Loller-Andersen and B. Gambäck, "Deep Learning-based Poetry Generation Given Visual Input," in ICCC, 2018, pp. 240-247.
[9] K. B. Lee, S. Cheon, and C. O. Kim, "A convolutional neural network for fault classification and diagnosis in semiconductor manufacturing processes," IEEE Transactions on Semiconductor Manufacturing, vol. 30, no. 2, pp. 135-142, 2017.
[10] T. Wang, Y. Chen, M. Qiao, and H. Snoussi, "A fast and robust convolutional neural network-based defect detection model in product quality control," The International Journal of Advanced Manufacturing Technology, vol. 94, no. 9, pp. 3465-3471, 2018.
[11] M. Bojarski et al., "End to end learning for self-driving cars," arXiv preprint arXiv:1604.07316, 2016.
[12] M. Al-Qizwini, I. Barjasteh, H. Al-Qassab, and H. Radha, "Deep learning algorithm for autonomous driving using googlenet," in 2017 IEEE Intelligent Vehicles Symposium (IV), 2017: IEEE, pp. 89-96.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, no. 6, pp. 84-90, 2017.
[14] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
[15] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[16] J. Wan, X. Li, H.-N. Dai, A. Kusiak, M. Martínez-García, and D. Li, "Artificial-intelligence-driven customized manufacturing factory: key technologies, applications, and challenges," Proceedings of the IEEE, vol. 109, no. 4, pp. 377-398, 2020.
[17] S. Liu, L. Liu, J. Tang, B. Yu, Y. Wang, and W. Shi, "Edge computing for autonomous driving: Opportunities and challenges," Proceedings of the IEEE, vol. 107, no. 8, pp. 1697-1716, 2019.
[18] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, "Edge computing: Vision and challenges," IEEE internet of things journal, vol. 3, no. 5, pp. 637-646, 2016.
[19] "tinyML Resources." Arm. https://www.arm.com/zh-TW/campaigns/arm-tinyml (accessed.
[20] E. Flamand et al., "GAP-8: A RISC-V SoC for AI at the Edge of the IoT," in 2018 IEEE 29th International Conference on Application-specific Systems, Architectures and Processors (ASAP), 2018: IEEE, pp. 1-4.
[21] "tinyml summit." tinyML Fountation. https://www.tinyml.org/ (accessed.
[22] H. Alshazly, C. Linse, E. Barth, and T. Martinetz, "Handcrafted versus CNN features for ear recognition," Symmetry, vol. 11, no. 12, p. 1493, 2019.
[23] S. Bubeck and M. Sellke, "A universal law of robustness via isoperimetry," Advances in Neural Information Processing Systems, vol. 34, pp. 28811-28822, 2021.
[24] L. Deng, G. Li, S. Han, L. Shi, and Y. Xie, "Model compression and hardware acceleration for neural networks: A comprehensive survey," Proceedings of the IEEE, vol. 108, no. 4, pp. 485-532, 2020.
[25] B. Widrow, I. Kollar, and M.-C. Liu, "Statistical theory of quantization," IEEE Transactions on instrumentation and measurement, vol. 45, no. 2, pp. 353-361, 1996.
[26] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning filters for efficient convnets," arXiv preprint arXiv:1608.08710, 2016.
[27] J. Ye, X. Lu, Z. Lin, and J. Z. Wang, "Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers," arXiv preprint arXiv:1802.00124, 2018.
[28] X. Ding, G. Ding, Y. Guo, J. Han, and C. Yan, "Approximated oracle filter pruning for destructive cnn width optimization," in International Conference on Machine Learning, 2019: PMLR, pp. 1607-1616.
[29] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, "Play and prune: Adaptive filter pruning for deep model compression," arXiv preprint arXiv:1905.04446, 2019.
[30] Z. Wang, F. Li, G. Shi, X. Xie, and F. Wang, "Network pruning using sparse learning and genetic algorithm," Neurocomputing, vol. 404, pp. 247-256, 2020.
[31] S. Katoch, S. S. Chauhan, and V. Kumar, "A review on genetic algorithm: past, present, and future," Multimedia tools and applications, vol. 80, pp. 8091-8126, 2021.
[32] Y. Li, K. Adamczewski, W. Li, S. Gu, R. Timofte, and L. Van Gool, "Revisiting random channel pruning for neural network compression," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 191-201.
[33] W.-C. Yeh, "Novel swarm optimization for mining classification rules on thyroid gland data," Information Sciences, vol. 197, pp. 65-76, 2012.
[34] W.-C. Yeh, "New parameter-free simplified swarm optimization for artificial neural network training and its application in the prediction of time series," IEEE Transactions on Neural Networks and Learning Systems, vol. 24, no. 4, pp. 661-665, 2013.
[35] P. Micikevicius et al., "Mixed precision training," arXiv preprint arXiv:1710.03740, 2017.
[36] X. Jia et al., "Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes," arXiv preprint arXiv:1807.11205, 2018.
[37] O. Kuchaiev et al., "Mixed-precision training for nlp and speech recognition with openseq2seq," arXiv preprint arXiv:1805.10387, 2018.
[38] D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex," The Journal of physiology, vol. 160, no. 1, p. 106, 1962.
[39] K. Fukushima and S. Miyake, "Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition," in Competition and cooperation in neural nets: Springer, 1982, pp. 267-285.
[40] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[41] C. Szegedy et al., "Going deeper with convolutions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1-9.
[42] M.-C. Popescu, V. E. Balas, L. Perescu-Popescu, and N. Mastorakis, "Multilayer perceptron and neural networks," WSEAS Transactions on Circuits and Systems, vol. 8, no. 7, pp. 579-588, 2009.
[43] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun, "What is the best multi-stage architecture for object recognition?," in 2009 IEEE 12th international conference on computer vision, 2009: IEEE, pp. 2146-2153.
[44] F. Rosenblatt, "The perceptron: a probabilistic model for information storage and organization in the brain," Psychological review, vol. 65, no. 6, p. 386, 1958.
[45] W. Luo, Y. Li, R. Urtasun, and R. Zemel, "Understanding the effective receptive field in deep convolutional neural networks," Advances in neural information processing systems, vol. 29, 2016.
[46] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818-2826.
[47] S. Basodi, C. Ji, H. Zhang, and Y. Pan, "Gradient amplification: An efficient way to train deep neural networks," Big Data Mining and Analytics, vol. 3, no. 3, pp. 196-207, 2020.
[48] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in International conference on machine learning, 2015: PMLR, pp. 448-456.
[49] D. Balduzzi, M. Frean, L. Leary, J. Lewis, K. W.-D. Ma, and B. McWilliams, "The shattered gradients problem: If resnets are the answer, then what is the question?," in International Conference on Machine Learning, 2017: PMLR, pp. 342-350.
[50] M. Denil, B. Shakibi, L. Dinh, M. A. Ranzato, and N. De Freitas, "Predicting parameters in deep learning," Advances in neural information processing systems, vol. 26, 2013.
[51] Y. LeCun, J. Denker, and S. Solla, "Optimal brain damage," Advances in neural information processing systems, vol. 2, 1989.
[52] S. Han, J. Pool, J. Tran, and W. Dally, "Learning both weights and connections for efficient neural network," Advances in neural information processing systems, vol. 28, 2015.
[53] T. Gale, E. Elsen, and S. Hooker, "The state of sparsity in deep neural networks," arXiv preprint arXiv:1902.09574, 2019.
[54] S. Han, H. Mao, and W. J. Dally, "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," arXiv preprint arXiv:1510.00149, 2015.
[55] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," arXiv preprint arXiv:1803.03635, 2018.
[56] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, "A tutorial on the cross-entropy method," Annals of operations research, vol. 134, no. 1, pp. 19-67, 2005.
[57] H. Robbins and S. Monro, "A stochastic approximation method," The annals of mathematical statistics, pp. 400-407, 1951.
[58] Y. Chauvin, "A back-propagation algorithm with optimal use of hidden units," Advances in neural information processing systems, vol. 1, 1988.
[59] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, "Pruning convolutional neural networks for resource efficient inference," arXiv preprint arXiv:1611.06440, 2016.
[60] S. Qiao, Z. Lin, J. Zhang, and A. L. Yuille, "Neural rejuvenation: Improving deep network training by enhancing computational resource utilization," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 61-71.
[61] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, "Learning efficient convolutional networks through network slimming," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2736-2744.
[62] N. Burgess, J. Milanovic, N. Stephens, K. Monachopoulos, and D. Mansell, "Bfloat16 processing for neural networks," in 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), 2019: IEEE, pp. 88-91.
[63] R. Poli, J. Kennedy, and T. Blackwell, "Particle swarm optimization," Swarm intelligence, vol. 1, no. 1, pp. 33-57, 2007.
[64] W.-C. Yeh, W.-W. Chang, and Y. Y. Chung, "A new hybrid approach for mining breast cancer pattern using discrete particle swarm optimization and statistical method," Expert Systems with Applications, vol. 36, no. 4, pp. 8204-8211, 2009.
[65] C.-L. Huang, "A particle-based simplified swarm optimization algorithm for reliability redundancy allocation problems," Reliability Engineering & System Safety, vol. 142, pp. 221-230, 2015.
[66] C.-M. Lai, W.-C. Yeh, and C.-Y. Chang, "Gene selection using information gain and improved simplified swarm optimization," Neurocomputing, vol. 218, pp. 331-338, 2016.
[67] A. Kumar, A. M. Shaikh, Y. Li, H. Bilal, and B. Yin, "Pruning filters with L1-norm and capped L1-norm for CNN compression," Applied Intelligence, vol. 51, no. 2, pp. 1152-1160, 2021.
[68] Y. C. Ho and X. Cao, "Perturbation analysis and optimization of queueing networks," Journal of optimization theory and Applications, vol. 40, no. 4, pp. 559-582, 1983.
[69] O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra, "Matching networks for one shot learning," Advances in neural information processing systems, vol. 29, 2016.
[70] M. Lin et al., "Hrank: Filter pruning using high-rank feature map," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1529-1538.
[71] S. Gao, F. Huang, W. Cai, and H. Huang, "Network pruning via performance maximization," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9270-9280.
[72] C. Yang and H. Liu, "Channel pruning based on convolutional neural network sensitivity," Neurocomputing, vol. 507, pp. 97-106, 2022.
[73] Z. Huang and N. Wang, "Data-driven sparse structure selection for deep neural networks," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 304-320.
[74] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, "Amc: Automl for model compression and acceleration on mobile devices," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 784-800.