本論文針對深度學習網路提出了一個全新的在線測試與修復架構,並提出了一個相對應的系統化設計與分析方法。在實驗中,我們用了 10 種不同的深度網路模型,包括 VGG11/16/19、ResNet18/34/50、以及 DenseNet121/161/169 來模擬和驗證我們提出的架構和方法。實驗結果顯示,我們所提出的架構和方法能夠在不同的神經網路錯誤分佈下,有效地提升因錯誤而下降的模型準確度。同時,我們也發現在有錯誤的情況下,基於 ReLU 的模型比基於 SELU 的模型有較高的錯誤容忍度。另外,基於 ReLU 的 DenseNet169 分類器在錯誤尺寸為 8 且錯誤數量多達 512 時,其準確度下降率只有-1.78%。基於本論文所提出的架構及方法,我們能夠有效率地在不同的錯誤分佈下,找出針對特定深度網路模型的高可靠性在線測試與修復架構。

In this thesis, we present a novel systematic evaluation flow for online test and repair of CNN-based MLP classifiers. Also, we propose an enhanced scheme for the classifiers, which are able to achieve smaller accuracy drops in different error distributions. To demonstrate the proposed evaluation flow and the enhanced scheme, we experimented with 10 different CNN-based classifiers, including VGG11/13/16/19, ResNet18/34/50, and DenseNet121/161/169, whose numbers of layers range from 11 to 169. The experimental results show that the enhanced ReLU-based VGG13 classifier can achieve only -2.94% accuracy drop when error size is 8 and error counts in the last hidden layer are up to 2,048. In addition, based on the proposed systematic flow, we found that the ReLU-based DenseNet169 classifier can achieve only -1.78% accuracy drop natively when error size is 8 and error counts are up to 512.

Abstract i
Contents ii
List of Figures..............................................................................................................iv
List of Tables................................................................................................................vi
List of Equations........................................................................................................vii
Chapter 1 Introduction..............................................................................................1
Chapter 2 Fundamentals of the Artificial Neural Network ...................................5
2.1 Neurons, Activation Functions, and MLP...........................................................5
2.2 Convolution Operation and Pooling....................................................................8
Chapter 3 MLP Error Definition............................................................................ 11
Chapter 4 The Proposed Online Test and Repair Scheme ...................................15
4.1 Additional Layers for Online Test and Repair ..................................................15
4.2 Training and Testing of The Proposed Scheme.................................................19
Chapter 5 The Proposed Systematic Evaluation Flow .........................................20
Chapter 6 Experimental Results.............................................................................23
6.1 Experiment Setup ..............................................................................................23
6.2 Experiments of VGG11/13/16/19 .....................................................................24
6.2.1 Output Value Distributions of VGG-based Classifiers................................24
6.2.2 Accuracy Evaluation of Original and Enhanced VGG-based Classifiers....25
6.3 Experiments of ResNet 18/34/50 ......................................................................33
6.3.1 Output Value Distributions of ResNet-based Classifiers.............................33
6.3.2 Accuracy Evaluation of Original and Enhanced ResNet-based Classifiers 34
6.4 Experiments of DenseNet 121/161/169 ............................................................41
6.4.1 Output Value Distributions of DesNet-based Classifiers ............................41
6.4.2 Accuracy Evaluation of Original and Enhanced DesNet-based Classifiers 42

iii

6.5 Overhead Analysis ............................................................................................48
Chapter 7 Conclusions and Future Work..............................................................50
7.1 Conclusions.......................................................................................................50
7.2 Future Work.......................................................................................................50
Bibliography ...............................................................................................................52

[1] M. Crawford, T. M. Khoshgoftaar, J. D. Prusa, A. N. Richter, and H. A. Najada,
"Survey of review spam detection using machine learning techniques", Journal
of Big Data, vol. 2, no. 23, pp. 1–24, Oct. 2015.
[2] Y. Deldjoo, M. Elahi, P. Cremonesi, F. Garzotto, P. Piazzolla, and M. Quadrana,
"Content-based video recommendation system based on stylistic visual features",
Journal on Data Semantics, vol. 5, no. 2, pp. 99–113, Feb. 2016.
[3] C. Amrit, T. Paauw, R. Aly, and M. Lavric, "Identifying child abuse through text
mining and machine learning", Expert Systems with Applications, vol. 88, 402–
18, Dec. 2017.
[4] M. Pak and S. Kim, "A review of deep learning in image recognition", in Proc.
4th International Conference on Computer Applications and Information
Processing Technology, Kuta Bali, Aug. 2017.
[5] E. Hossain, I. Khan, F. Un-Noor, S. S. Sikander, and M. S. H. Sunny, "Application
of big data and machine learning in smart grid, and associated security concerns:
a review", IEEE Access, vo. 7, pp. 13960–188, Feb. 2019.
[6] M. B. Rozenwald, A. A. Galitsyna, G. V. Sapunov, E. E. Khrameeva, and M. S.
Gelfand, "A machine learning framework for the prediction of chromatin folding
in Drosophila using epigenetic features", PeerJ Computer Science, vol. 6, pp. 1–
21, Nov. 2020.
[7] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, "A survey of deep
neural network architectures and their applications", Neurocomputing, vol. 234,
pp. 11–26, Apr. 2017.
[8] S. Pouyanfar, S. Sadiq, Y. Yan, H. Tian, Y. Tao, M. P. Reyes, M.-L. Shyu, S.-C.
Chen, and S. S. Iyengar, "A survey on deep learning: algorithms, techniques, and
applications", ACM Computing Surveys, vol. 51, no. 92, pp. 1–36, Sep. 2019.
[9] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J.
Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, "Review of deep
learning: concepts, CNN architectures, challenges, applications, future
directions" Journal of Big Data, vol. 8, no. 53, pp. 1–74, Mar. 2021.
[10] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, "A survey of the recent
architectures of deep convolutional neural networks", Artificial Intelligence Review, vol. 53, pp. 5455–5516, Apr. 2020.
[11] R. I. Hasan, S. M. Yusuf, and L. Alzubaidi, "Review of the state of the art of deep
learning for plant diseases: a broad analysis and discussion", Plants (Basel,
Switzerland), vol. 9, pp. 1–25, Oct. 2020,
[12] Y.Xiao, Z. Tian, J. Yu, Y. Zhang, S. Liu, S. Du, and X. Lan, "A review of object
detection based on deep learning", Multimedia Tools and Applications, vol. 79,
no. 33, pp. 23729–91, Apr. 2020.
[13] J. Ker, L. Wang, J. Rao and T. Lim, "Deep learning applications in medical image
analysis", IEEE Access, vol. 6, pp. 9375–89, Dec. 2017.
[14] Y. LeCun, L. Jackel, L. Bottou, C. Cortes, J. Denker, H. Drucker, I. Guyon, U.
Muller, E. Sackinger, P. Simard, and V. Vapnik, "Learning algorithms for
classification: a comparison on handwritten digit recognition", Neural networks:
The statistical mechanics perspective, pp. 261–276, 1995.
[15] A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep
convolutional neural networks", Neural Information Processing Systems, vol. 25,
pp. 1–9, Jan. 2012.
[16] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A.
Karpathy, A. Khosla, M. Bernstein, A. C. Berg, F.-F. Li, "ImageNet Large Scale
Visual Recognition Challenge", arXiv, Sep. 2014. Available:
https://arxiv.org/pdf/1409.0575.pdf
[17] B. Xu, N. Wang, T. Chen, and M. Li, "Empirical evaluation of rectified activations
in convolutional network", arXiv, Nov. 2015. Available:
https://arxiv.org/pdf/1505.00853.pdf
[18] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "R.
Dropout: a simple way to prevent neural networks from overfitting", The Journal
of Machine Learning Research, vol. 15, pp. 1929–58, Jan. 2014.
[19] G. E. Dahl, T. N. Sainath, G. Hinton, "Improving deep neural networks for
LVCSR using rectified linear units and drop‐out", in Proc. 2013 IEEE
International Conference on Acoustics, Speech and Signal Processing, Vancouver,
May 2013.
[20] M. D. Zeiler and R. Fergus, "Stochastic pooling for regularization of deep
convolutional neural networks", arXiv, Jan. 2013. Available:
https://arxiv.org/abs/1301.3557
[21] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V.
Vanhoucke, and A. Rabinovich, "Going deeper with convolutions", in Proc. 2015
IEEE Conference on Computer Vision and Pattern Recognition, Boston, June
2015.
[22] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, "Rethinking the
Inception Architecture for Computer Vision", in Proc. 2016 IEEE Conference on
Computer Vision and Pattern Recognition, Las Vegas, June 2016.

[23] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, "Inception-v4, Inception-
ResNet and the Impact of Residual Connections on Learning", arXiv, Aug. 2016.

Available: https://arxiv.org/pdf/1602.07261.pdf

[24] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-
Scale Image Recognition", arXiv, Sep. 2014. Available:

https://arxiv.org/pdf/1409.1556.pdf
[25] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image
recognition", in Proc. 2016 IEEE Conference on Computer Vision and Pattern
Recognition, Las Vegas, June 2016.
[26] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, F.-F. Li, A. Yuille, J.
Huang, and K. Murphy, "Progressive Neural Architecture Search", arXiv, Dec.
2017. Available: https://arxiv.org/pdf/1712.00559.pdf
[27] J. Hu, L. Shen and G. Sun, "Squeeze-and-Excitation Networks", in Proc. 2018
IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake
City, June 2018.
[28] G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger, "Densely connected
convolutional networks", in Proc. 2017 IEEE Conference on Computer Vision
and Pattern Recognition, Honolulu, July 2017.
[29] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, "Self-Normalizing
Neural Networks", arXiv, Jun. 2017. Available:
https://arxiv.org/pdf/1706.02515.pdf
[30] H. Gholamalinezhad, and H. Khosravi, "Pooling Methods in Deep Neural
Networks, a Review", arXiv, Sep. 2020. Available:
https://arxiv.org/ftp/arxiv/papers/2009/2009.07485.pdf
[31] A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images", 2009.
Available: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
[32] C. M. Bishop, "Pattern Recognition and Machine Learning", Springer, 2006.