近年來Vision Transformer模型在各種電腦視覺任務中表現卓越，例如圖像分類和物體偵測，現在已經成為最強大的深度學習模型之一。但是若想要將這些模型應用在一些更下游的領域中，我們必須對這些模型的可靠性進行更深入的研究，尤其像是自動駕駛車輛和醫學影像這樣注重安全性和可靠性的領域。同時隨著科技的發展，現在電腦系統當中的電子元件的大小也持續的縮減，使得這些元件更容易受到所處環境中的宇宙射線的影響，發生硬體的暫態失效，也就是所謂的軟錯誤。考慮到如果在注重安全性和可靠性的應用中，模型的運作失常可能帶來非常嚴重的後果，再加上軟錯誤發生的頻率不斷增加，我們決定針對Vision Transformer模型進行全面的可靠性分析，並設計方法來減輕這些軟錯誤在模型的推論中帶來的影響。在本論文中，我們透過針對模型參數的故障注入進行軟錯誤的模擬，分析在軟錯誤存在的情況下Vision Transformer模型的可靠性。我們也將分析擴展到各種Vision Transformer模型的不同變體上，並另外分析故障注入對於量化過的Vision Transformer模型與其變體造成的影響。除了以上的分析外，我們還提出一個針對Vision Transformer模型的簡單且非常有效的故障緩解方法。這種故障緩解技術可以將這些Vision Transformer模型在ImageNet-1K上的Top-1準確度在有軟錯誤存在的狀況下恢復到正常的水準。

Vision Transformer models have become a powerful class of deep learning models, excelling in various computer vision tasks. While their success in image classification and object detection is evident, their integration into safety-critical domains like autonomous vehicles and medical imaging necessitates a robust understanding of their safety and reliability. This is particularly crucial due to the rising vulnerability of modern computing systems to transient hardware faults, namely soft errors. In this thesis, we analyze the reliability of Vision Transformer models in the presence of soft errors by employing parameter fault injection for simulation. Our analysis extends to various variants of Vision Transformer models, including an additional examination of the impact of fault injection on quantized Vision Transformer models and their variants. Beyond the analyses mentioned, we propose a simple yet highly effective fault mitigation technique for Vision Transformer models. This fault mitigation technique can restore the Top-1 accuracy of Vision Transformer models on ImageNet-1K to nearly original levels, even in the presence of soft errors.

1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Our Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Preliminaries 4
2.1 Vision Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 Self-attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.2 Multi-head Self-attention . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Neural Network Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Post-Training Quantization (PTQ) . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Quantization-Aware Training (QAT) . . . . . . . . . . . . . . . . . . . 6
2.3 Reliability Analysis of Neural Network . . . . . . . . . . . . . . . . . . . . . 7
2.3.1 Fault Injection Methods . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Methodology 8
3.1 Fault Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Fault Injection for Reliability Analysis . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.2 Vision Transformer Models . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.3 Fault Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.4 Fault Injection Framework . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Fault Mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3.1 Range Restriction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.2 Range Restriction Clip . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4 Experimental Results 17
4.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Reliability Evaluation for Unquantized Models . . . . . . . . . . . . . . . . . 17
4.3 The Impact of Soft Errors on Unquantized Models of Varying Sizes . . . . . . 19
4.4 Reliability Evaluation for Quantized Model . . . . . . . . . . . . . . . . . . . 19
4.5 The Effectiveness of Proposed Fault Mitigation Technique . . . . . . . . . . . 22
5 Conclusion 27
References 28

[1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proceedings of International Conference on Learning Representations, 2021.
[2] D. Parikh, “Recognizing jumbled images: The role of local and global information in image classification,” in Proceedings of International Conference on Computer Vision, pp. 519–526, 2011.
[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. U. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of Neural Information Processing Systems, pp. 5999–6009, 2017.
[4] M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. Van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295, 2021.
[5] M. H. Ahmadilivani, M. Taheri, J. Raik, M. Daneshtalab, and M. Jenihhin, “A systematic literature review on hardware reliability assessment methods for deep neural networks,” ACM Computing Surveys, vol. 56, no. 6, pp. 141:1–141:39, 2024.
[6] A. Ruospo, L. M. Luza, A. Bosio, M. Traiola, L. Dilillo, and E. Sanchez, “Pros and cons of fault injection approaches for the reliability assessment of deep neural networks,” in Proceedings of IEEE Latin American Test Symposium, pp. 87–91, 2021.
[7] G. Bolt, “Fault models for artificial neural networks,” in Proceedings of IEEE International Joint Conference on Neural Networks, pp. 1373–1378, 1991.
[8] C. Liu, Z. Gao, S. Liu, X. Ning, H. Li, and X. Li, “Fault-tolerant deep learning: A hierarchical perspective,” arXiv preprint arXiv:2204.01942, 2022.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
[10] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & distillation through attention,” in Proceedings of International Conference on Machine Learning, pp. 10347–10357, 2021.
[11] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of International Conference on Computer Vision, pp. 9992–10002, 2021.
[12] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5987–5995, 2017.
[13] M. Hasan and B. Ray, “Tolerance of deep neural network against the bit error rate of nand flash memory,” in Proceedings of IEEE International Reliability Physics Symposium, pp. 814–817, 2019.
[14] M. M. Hasan and B. Ray, “Reliability of nand flash memory as a weight storage device of artificial neural network,” IEEE Transactions on Device and Materials Reliability, vol. 20, no. 3, pp. 596–603, 2020.
[15] B. Sangchoolie, K. Pattabiraman, and J. Karlsson, “One bit is (not) enough: An empirical study of the impact of single and multiple bit-flip errors,” in Proceedings of IEEE International Conference on Dependable Systems and Networks, pp. 97–108, 2017.
[16] Z. Chen, G. Li, K. Pattabiraman, and N. DeBardeleben, “Binfi: An efficient fault injector for safety-critical machine learning systems,” in Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 69:1–69:23, 2019.
[17] Z. Chen, G. Li, and K. Pattabiraman, “A low-cost fault corrector for deep neural networks through range restriction,” in Proceedings of IEEE International Conference on Dependable Systems and Networks, pp. 1–13, 2021.
[18] Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou, “Fq-vit: Post-training quantization for fully quantized vision transformer,” in Proceedings of International Joint Conference on Artificial Intelligence, pp. 1173–1179, 2022.
[19] I. S. Haque and V. S. Pande, “Hard data on soft errors: A large-scale assessment of real-world error rates in gpgpu,” in Proceedings of IEEE International Conference on Cluster, Cloud and Grid Computing, pp. 691–696, 2010.