目前多數研究僅針對分心、疲勞進行個別的研究，但僅討論這兩點肇事因素無法完全的解釋及避免所有車禍因素。且目前研究應用生理資訊大多屬於後續分析，並未將生理資訊做到即時處理的應用，也缺乏整合電腦視覺與生理資訊的智能警示系統。為了解決這一問題並增加實務上與生理資訊即時的應用，本研究使用YOLOv5電腦視覺進行偵測，YOLOv5除了考量分心與疲勞外，也列入情緒這項因素(Alkinani et al., 2020)，並透過生理資訊(EEG、HRV)的即時輔助判斷，建立智能警示系統，以更全面地解釋車禍因素。同時也解決過往駕駛安全研究中缺乏整合分心、疲勞和情緒的不足。然而，目前公開資料集缺乏整合分心、疲勞和情緒之資料集應用於汽車安全領域，因此本研究也在駕駛模擬環境中自行建立一泛化程度良好之資料集進行訓練。研究中使用深度學習之YOLOv5對駕駛者臉部及肢體動作特徵的影像，並透過生理資訊的即時處理，輔助偵測駕駛情緒與駕駛疲勞，建立個人化的智能駕駛警示系統，使用警鈴方式給予駕駛者更具實用性的車載警示系統。本研究透過一系列的實驗設計分析30位受試者的駕駛行為與問卷結果，探討本研究提出之智能警示系統是否有助於提升駕駛安全性，並分析三大不安全駕駛行為之嚴重性。從結果可以得知，本研究提出之智能警示系統準確率達98.4%，同時在偵測速度上也最高也達125FPS/秒。最後，本研究提出電腦視覺之不安全駕駛行為檢測輔以生理資訊的及時應用，致使該系統在具有快速的判斷的優勢下，仍然擁有精確的準確率，同時也解決了過往研究在準確率、個人化與實用性不足的問題。

At present, most studies only conduct individual studies on distraction and fatigue, but only discussing these two accident factors cannot fully explain and avoid all car accident factors. In addition, most of the current research and application of physiological information belongs to the follow-up analysis, and the application of physiological information is not realized in real time, and there is also a lack of intelligent warning system integrating computer vision and physiological information. In order to solve this problem and increase the application of real-time practical and physiological information, this study uses YOLOv5 computer vision for detection. In addition to considering distraction and fatigue, YOLOv5 also includes emotion as a factor (Alkinani et al., 2020), and through the real-time auxiliary judgment of physiological information (EEG, HRV), an intelligent warning system is established to more comprehensively explain the factors of traffic accidents. It also addresses the lack of integration of distraction, fatigue, and emotion in previous driving safety research. However, there is currently a lack of public datasets that integrate distraction, fatigue, and emotion for use in the field of automotive safety. Therefore, this study also builds a well-generalized dataset for training in a driving simulation environment. In the research, YOLOv5 of deep learning is used to image the driver's facial and body movement characteristics, and through the real-time processing of physiological information, it can assist in detecting driving emotions and driving fatigue, and establish a personalized intelligent driving warning system. A more practical vehicle warning system for the driver. This study analyzed the driving behaviors and questionnaire results of 30 subjects through a series of experimental designs to explore whether the intelligent warning system proposed in this study can help improve driving safety, and to analyze the severity of the three major unsafe driving behaviors. It can be seen from the results that the accuracy rate of the intelligent warning system proposed in this study is 98.4%, and the detection speed is also up to 125FPS/sec. Finally, this study proposes that computer vision for unsafe driving behavior detection is supplemented by the timely application of physiological information, so that the system still has accurate accuracy despite the advantages of rapid judgment. The problem of lack of personalization and practicality.

摘要 ii
誌謝 i
1 緒論 2
2 文獻回顧 5
2.1 生理資訊與統計問卷量表於駕駛安全檢測 5
2.2 人工智能在駕駛安全的應用 7
2.3 小節 11
3 研究方法 12
3.1 資料蒐集與前處理 13
3.1.1 資料蒐集與標註 13
3.1.2 資料增強 14
3.2 模型訓練與驗證 16
3.2.1 YOLOv5模型訓練 16
3.2.2 YOLOv5模型驗證 19
3.3 生理資訊檢測 22
3.3.1 主觀檢測量表 22
3.3.2 客觀儀器測量 26
3.4 整合智能駕駛警示系統 29
3.5 驗證 31
3.5.1 實驗設計 31
3.5.2 資料分析 32
4 案例實證 37
4.1 資料集使用與資料前處理 37
4.2 YOLOv5模型訓練 38
4.3 驗證 45
4.3.1 使用者生理資訊分析 50
4.3.2 使用者體驗分析 54
4.3.3 智能警示系統有效性分析 59
4.3.4 分心、疲勞與情緒嚴重性分析 60
4.3.5 駕駛行為統計分析 63
4.4 討論 63
5 結論 68

Alkinani, M. H., Khan, W. Z., & Arshad, Q. (2020). Detecting human driver inattentive and aggressive driving behavior using deep learning: Recent advances, requirements and open challenges. Ieee Access, 8, 105008-105030.
Appelhans, B. M., & Luecken, L. J. (2006). Heart rate variability as an index of regulated emotional responding. Review of general psychology, 10(3), 229-240.
Ataş, K., & Vural, R. A. (2021, October). Detection of Driver Distraction using YOLOv5 Network. In 2021 2nd Global Conference for Advancement in Technology (GCAT) (pp. 1-6). IEEE.
Ayeni, Aanuolupo et al. 2019. “Self-Reported Sleepiness in the Context of Fitness-to-Drive.” Sleep and Breathing 23(4): 1227–32.
Baheti, Bhakti, SanjayTalbar, and SuhasGajre. 2020. “Towards Computationally Efficient and Realtime Distracted Driver Detection with MobileVGG Network.” IEEE Transactions on Intelligent Vehicles 5(4): 565–74.
Bazar, K. A., Yun, A. J., & Lee, P. Y. (2004). Debunking a myth: neurohormonal and vagal modulation of sleep centers, not redistribution of blood flow, may account for postprandial somnolence. Medical hypotheses, 63(5), 778-782.
Brooke, J. (1996). Sus: a “quick and dirty’usability. Usability evaluation in industry, 189(3).
Bodavarapu, P. N. R., & Srinivas, P. S. (2021). An optimized neural network model for facial expression recognition over traditional deep neural networks. International Journal of Advanced Computer Science and Applications, 12(7).
Bochkovskiy, Alexey, Chien-YaoWang, and Hong-Yuan MarkLiao. 2020. “YOLOv4: Optimal Speed and Accuracy of Object Detection.”view on: http://arxiv.org/abs/2004.10934.
Chaturvedi, A., Guru, A., Kumar, N., Lin, L. Y., Wei, D. Y., Sheng, L. K., & Yee, L. H. (2021). Postprandial Somnolence and its awareness among the Medical Undergraduate Students: A crosssectional study. Bangladesh Journal of Medical Science, 20(4), 826-832.
Chen, L. L., Zhang, A., & Lou, X. G. (2019). Cross-subject driver status detection from physiological signals based on hybrid feature selection and transfer learning. Expert Systems with Applications, 137, 266-280.
Chiu, M-C. and Chuang, K-H. (2021) "Applying Transfer Learning to Achieve Precision Marketing in an Omni-Channel System – A Case Study of a Sharing Kitchen Platform," International Journal of Production Research, 59(24), 7594-7609.
Chiu, M. C., & Chen, T. M. (2021). Applying Data Augmentation and Mask R-CNN-Based Instance Segmentation Method for Mixed-Type Wafer Maps Defect Patterns Classification. IEEE Transactions on Semiconductor Manufacturing, 34(4), 455-463.
Chiu, M. C., Tsai, H. Y., & Chiu, J. E. (2022). A novel directional object detection method for piled objects using a hybrid region-based convolutional neural network. Advanced Engineering Informatics, 51, 101448.
Chiu, M. C., & Ko, L. W. (2017). Develop a personalized intelligent music selection system based on heart rate variability and machine learning. Multimedia Tools and Applications, 76(14), 15607-15639.
Colyer, S. L., Evans, M., Cosker, D. P., & Salo, A. I. (2018). A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system. Sports medicine-open, 4(1), 24.
de Naurois, C. J., Bourdin, C., Bougard, C., & Vercher, J. L. (2018). Adapting artificial neural networks to a specific driver enhances detection and prediction of drowsiness. Accident Analysis & Prevention, 121, 118-128.
Dong, Bing Ting, and Huei YungLin. 2021. “An On-Board Monitoring System for Driving Fatigue and Distraction Detection.” Proceedings of the IEEE International Conference on Industrial Technology 2021-March: 850–55.
Dwivedi, S. A., Attry, A., & Singla, K. (2022). Leveraging Transfer Learning for Driver Drowsiness Detection. In Advances in Data and Information Sciences (pp. 603-611). Springer, Singapore.
Deffenbacher, J. L., Oetting, E. R., & Lynch, R. S. (1994). Development of a driving anger scale. Psychological reports, 74(1), 83-91.
Fanfulla, Francesco et al. 2021. “Determinants of Sleepiness at Wheel and Missing Accidents in Patients With Obstructive Sleep Apnea.” Frontiers in Neuroscience 15(April): 1–9.
Feng, Zhihua, XinLuo, TaoYang, and KenjiKita. 2018. “An Object Detection System Based on YOLOv2 in Fashion Apparel.” 2018 IEEE 4th International Conference on Computer and Communications, ICCC 2018: 1532–36.
Guo, Feng et al. 2017. “The Effects of Age on Crash Risk Associated with Driver Distraction.” International Journal of Epidemiology 46(1): 258–65.
Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139-183). North-Holland.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 37(9), 1904-1916.
Hoddes, E., Zarcone, V., & Dement, W. (1972). Stanford sleepiness scale. Enzyklopädie der Schlafmedizin, 1184(1).
Horne, J., & Reyner, L. (1999). Vehicle accidents related to sleep: a review. Occupational and environmental medicine, 56(5), 289-294.
Iio, Kentaro, XiaoyuGuo, and DominiqueLord. 2021. “Examining Driver Distraction in the Context of Driving Speed: An Observational Study Using Disruptive Technology and Naturalistic Data.” Accident Analysis and Prevention 153(December 2020): 105983. view on: https://doi.org/10.1016/j.aap.2021.105983.
Islam, Md Motaharul et al. 2020. “An Algorithmic Approach to Driver Drowsiness Detection for Ensuring Safety in an Autonomous Car.” 2020 IEEE Region 10 Symposium, TENSYMP 2020 (June): 328–33.
Jiao, L., Zhang, F., Liu, F., Yang, S., Li, L., Feng, Z., & Qu, R. (2019). A Survey of Deep Learning-Based Object Detection. IEEE Access, 7, 128837-128868.
Johns, M. W. (1991). A new method for measuring daytime sleepiness: the Epworth sleepiness scale. sleep, 14(6), 540-545.
Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Zhang, C., Wang, Z., Wang, R., Wang X., et al., “T-cnn: Tubelets with convolutional neural networks for object detection from videos,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 10, pp. 2896–2907, 2018.
Kim, W., Choi, H. K., Jang, B. T., & Lim, J. (2017, October). Driver distraction detection using single convolutional neural network. In 2017 international conference on information and communication technology convergence (ICTC) (pp. 1203-1205). IEEE.
Kusuma, G. P., Jonathan, A. P. L., & Lim, A. P. (2020). Emotion recognition on fer-2013 face images using fine-tuned vgg-16. Advances in Science, Technology and Engineering Systems Journal, 5(6), 315-322.
Kulkarni, Ruturaj, ShrutiDhavalikar, and SonalBangar. 2018. “Traffic Light Detection and Recognition for Self Driving Cars Using Deep Learning.” Proceedings - 2018 4th International Conference on Computing, Communication Control and Automation, ICCUBEA 2018: 1–4.
Kwon, Sooyoung, HeejungKim, Gwang SukKim, and EunheeCho. 2019. “Fatigue and Poor Sleep Are Associated with Driving Risk among Korean Occupational Drivers.” Journal of Transport and Health 14(May): 100572. view on: https://doi.org/10.1016/j.jth.2019.100572.
Lattanzi, Emanuele, and ValerioFreschi. 2021. “Machine Learning Techniques to Identify Unsafe Driving Behavior by Means of In-Vehicle Sensor Data.” Expert Systems with Applications 176(May 2020): 114818. view on: https://doi.org/10.1016/j.eswa.2021.114818.
LaRocco, J., Le, M. D., & Paeng, D. G. (2020). A systemic review of available low-cost EEG headsets used for drowsiness detection. Frontiers in neuroinformatics, 42.
Lauer, R. T., Peckham, P. H., & Kilgore, K. L. (1999). "EEG‐based control of a hand grasp neuroprosthesis". Neuroreport, 10(8), 1767-1771.
Li, Zhaojian, ShanBao, IlyaV.Kolmanovsky, and XiangYin. 2018. “Visual-Manual Distraction Detection Using Driving Performance Indicators with Naturalistic Driving Data.” IEEE Transactions on Intelligent Transportation Systems 19(8): 2528–35.
Liang, H., Fu, W., & Yi, F. (2019, October). A survey of recent advances in transfer learning. In 2019 IEEE 19th international conference on communication technology (ICCT) (pp. 1516-1523). IEEE.
Luo, L., & Shi, J. (2020). Aberrant driving behaviors by tourists: a study of drivers in China. Accident Analysis & Prevention, 146, 105738.
Mahima, K. T.Y., and T. N.D.S.Ginige. 2021. “An Emotion Recognition Based Assistant for Vehicles.” 2021 International Conference on Computer Communication and Artificial Intelligence, CCAI 2021: 1–5.
Manikandan, A., & Sujith, M. (2021, August). A Novel System for Real Time Drowsiness Warning and Engine Ignition Authorization using Face Recognition. In 2021 Second International Conference on Electronics and Sustainable Communication Systems (ICESC) (pp. 1658-1663). IEEE.
Martinez-Maradiaga, D., & Meixner, G. (2017, November). Morpheus alert: A smartphone application for preventing microsleeping with a brain-computer-interface. In 2017 4th International Conference on Systems and Informatics (ICSAI) (pp. 137-142). IEEE.
Miwata, M., Tsuneyoshi, M., Tada, Y., Ikeda, M., & Barolli, L. (2021, July). Design of an intelligent driving support system for detecting distracted driving. In Conference on Complex, Intelligent, and Software Intensive Systems (pp. 377-382). Springer, Cham.
Moslemi, Negar, MohsenSoryani, and RezaAzmi. 2021. “Computer Vision‐based Recognition of Driver Distraction: A Review.” Concurrency and Computation: Practice and Experience.
Msonda, Pike, Sait AliUymaz, and Seda SoǧukpinarKaraaǧaç. 2020. “Spatial Pyramid Pooling in Deep Convolutional Networks for Automatic Tuberculosis Diagnosis.” Traitement du Signal 37(6): 1075–84.
Muhammad, K., Ullah, A., Lloret, J., Del Ser, J., & de Albuquerque, V. H. C. (2020). Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems, 22(7), 4316-4336.
Nguyen, D. L., Putro, M. D., & Jo, K. H. (2021, October). Distracted Driver Recognizer with Simple and Efficient Convolutional Neural Network for Real-time System. In 2021 21st International Conference on Control, Automation and Systems (ICCAS) (pp. 371-375). IEEE.
Orlovska, Julia et al. 2020. “Effects of the Driving Context on the Usage of Automated Driver Assistance Systems (ADAS) -Naturalistic Driving Study for ADAS Evaluation.” Transportation Research Interdisciplinary Perspectives 4.
Parasuraman, A., Berry, L., & Zeithaml, V. (2002). Refinement and reassessment of the SERVQUAL scale. Journal of retailing, 67(4), 114.
Putra, Agfianto Eko, CaturAtmaji, and Tifani GaluhUtami. 2017. “EEG-Based Microsleep Detector Using Microcontroller.” Proceedings of 2016 8th International Conference on Information Technology and Electrical Engineering: Empowering Technology for Better Future, ICITEE 2016.
Qin, B., Qian, J., Xin, Y., Liu, B., & Dong, Y. (2021). Distracted driver detection based on a CNN with decreasing filter size. IEEE Transactions on Intelligent Transportation Systems.
Rashmi, M., & Guddeti, R. M. R. (2020, January). Skeleton based Human Action Recognition for Smart City Application using Deep Learning. In 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS) (pp. 756-761). IEEE.
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779-788).
Redmon, Joseph, and AliFarhadi. 2017. “YOLO V2.0.” Cvpr2017 (April): 187–213. view on: http://www.worldscientific.com/doi/abs/10.1142/9789812771728_0012.
Redmon, J., & Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv preprint arXiv:1804.02767.
Regan, M A et al. 2013. “The Australian 400-Car Naturalistic Driving Study: Innovation in Road Safety Research and Policy.” Australasian Road Safety Research Policing Education Conference, 2013, Brisbane, Queensland, Australia (August). view on: https://trid.trb.org/view.aspx?id=1286886.
Ren, Peiming et al. 2020. “A Novel Squeeze YOLO-Based Real-Time People Counting Approach.” International Journal of Bio-Inspired Computation 16(2): 94–101.
Revaud, J., Almazán, J., Rezende, R. S., & Souza, C. R. D. (2019). Learning with average precision: Training image retrieval with a listwise loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5107-5116).
Roseborough, James E.W., Christine M.Wickens, and David L.Wiesenthal. 2021. “Retaliatory Aggressive Driving: A Justice Perspective.” Accident Analysis and Prevention 162(August): 106393. view on: https://doi.org/10.1016/j.aap.2021.106393.
Russakovsky, Olga et al. 2015. “ImageNet Large Scale Visual Recognition Challenge.” International Journal of Computer Vision 115(3): 211–52. view on: http://dx.doi.org/10.1007/s11263-015-0816-y.
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: a database and web-based tool for image annotation. International journal of computer vision, 77(1-3), 157-173.
Saif, A. S., & Mahayuddin, Z. R. (2020). Robust drowsiness detection for vehicle driver using deep convolutional neural network. International Journal of Advanced Computer Science and Applications, 11(10).
Sikander, G., & Anwar, S. (2018). Driver fatigue detection systems: A review. IEEE Transactions on Intelligent Transportation Systems, 20(6), 2339-2352.
Sumalatha, R., Sravani, C., & Supriya, M. (2021). Detection of Driver Distraction Using Convolutional Neural Network. In Innovations in Cyber Physical Systems (pp. 315-323). Springer, Singapore.
Suresh, Y., Khandelwal, R., Nikitha, M., Fayaz, M., & Soudhri, V. (2021, October). Driver Drowsiness Detection using Deep Learning. In 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC) (pp. 1526-1531). IEEE.
Talo, M., Baloglu, U. B., Yıldırım, Ö., & Acharya, U. R. (2019). Application of deep transfer learning for automated brain abnormality classification using MR images. Cognitive Systems Research, 54, 176-188.
Tan, Shilei, JunYan, ZiqiangJiang, and LiHuang. 2021. “Approach for Improving YOLOv5 Network with Application to Remote Sensing Target Detection.” Journal of Applied Remote Sensing 15(03).
Ugli, I. K. K., Hussain, A., Kim, B. S., Aich, S., & Kim, H. C. (2022, February). A Transfer Learning Approach for Identification of Distracted Driving. In 2022 24th International Conference on Advanced Communication Technology (ICACT) (pp. 420-423). IEEE.
Vu, T. H., Dang, A., & Wang, J. C. (2019). A deep neural network for real-time driver drowsiness detection. IEICE TRANSACTIONS on Information and Systems, 102(12), 2637-2641.
Wang, C. Y., Liao, H. Y. M., Wu, Y. H., Chen, P. Y., Hsieh, J. W., & Yeh, I. H. 2020. “CSPNet: A New Backbone That Can Enhance Learning Capability of CNN.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2020-June: 1571–80.
Wang, X., Guo, Y., Bai, C., Yuan, Q., Liu, S., & Han, J. (2020). Driver's Intention Identification With the Involvement of Emotional Factors in Two-Lane Roads. IEEE Transactions on Intelligent Transportation Systems, 22(11), 6866-6874.
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), 1063.
Wu, Tian Hao, Tong WenWang, and Ya QiLiu. 2021. “Real-Time Vehicle and Distance Detection Based on Improved YOLO v5 Network.” 2021 3rd World Symposium on Artificial Intelligence, WSAI 2021: 24–28.
Xiao, H., Li, W., Zeng, G., Wu, Y., Xue, J., Zhang, J., ... & Guo, G. (2022). On-Road Driver Emotion Recognition Using Facial Expression. Applied Sciences, 12(2), 807.
Yan, Jun Juh, Hang HongKuo, Ying FanLin, and Teh LuLiao. 2016. “Real-Time Driver Drowsiness Detection System Based on PERCLOS and Grayscale Image Processing.” Proceedings - 2016 IEEE International Symposium on Computer, Consumer and Control, IS3C 2016: 243–46.
Yoo, I., Kim, E. J., & Lee, J. H. (2015). Effects of chewing gum on driving performance as evaluated by the STISIM driving simulator. Journal of physical therapy science, 27(6), 1823-1825.
Young, Kristie. 2011. “Driver Distraction: A Review of the Current State-of-Knowledge.” Distracted Driving (September): 1–22.
Zahara, L., Musa, P., Wibowo, E. P., Karim, I., & Musa, S. B. (2020, November). The facial emotion recognition (FER-2013) dataset for prediction system of micro-expressions face using the convolutional neural network (CNN) algorithm based Raspberry Pi. In 2020 Fifth international conference on informatics and computing (ICIC) (pp. 1-9). IEEE.
Zhou, Fangbo, HuailinZhao, and ZhenNie. 2021. “Safety Helmet Detection Based on YOLOv5.” Proceedings of 2021 IEEE International Conference on Power Electronics, Computer Applications, ICPECA 2021: 6–11.
National Highway Traffic Safety Administration . (2019). Traffic Safety Facts Annual Report Tables . Retrieved from : https://cdan.nhtsa.gov/tsftables/tsfar.htm
Ultralytics. (2020). YOLOv5. Retrieved from : https://github.com/ultralytics/YOLOv5
Medicore (2015) SA-3000P clinical manual ver. 3.0. Retrieved June 8, 2015, from http://medi-core.com/download/HRV_clinical_manual_ver3.0.pdf
台灣交通部 . (2021). 交通事故統計快覽圖表 - 道安資訊查詢網.
https://roadsafety.tw/Dashboard/Custom?type=%E7%B5%B1%E8%A8%88%E5%BF%AB%E8%A6%BD