近年來火龍果在台灣的種植面積與產量逐年增加，相比於十年前皆有三倍以上的成長，並且隨著台灣精湛的農業改良技術，不僅有效地火龍果的產量提升，也培育出不同的品種提供消費者挑選，然而火龍果在等級上的分級仍然有待進步，目前市場上的分級除了外觀上損壞之外，主要還是以重量為分級指標，無法對於果實的成熟度與口感進行分級，因此本研究主要利用火龍果的外觀來判斷其成熟度之分級，減少時間與人力的成本並且創造更高的效益。同時，我們也開發一項火龍果成熟度即時分級系統，結合了目前火龍果農場所使用的重量分級選果機，將研究成果應用於實際場域。

首先透過網路攝影機拍攝火龍果果實影像來蒐集資料，接著透過物件偵測與資料預處理的技術去處理每張影像，最後將一張一張影像依序放入深度學習中的卷積神經網路進行模型的訓練。透過卷積神經網路進行火龍果外觀的特徵擷取後，接著搭配深度神經網路的分類功能，讓模型能夠預測出每個火龍果的成熟度。模型預測階段上，利用同樣的網路攝影機抓取火龍果的影像，將影像放入邊緣計算裝置上的火龍果成熟度即時分級系統，預測出來的結果會傳送到分級選果機上，並依照成熟度進行分級。本研究的模型會將成熟度預測分為三個等級，其準確率達百分之九四，且各個等級之F1-score也均達到九成以上。

In recent years, planted area and production of dragonfruits in Taiwan increased year by year, both of them are three times above than ten years ago. As the improvement of agricultural research and extension, it not only increases production effectively, but also develops lots of brand-new species which provides consumers to purchase. However, grading on dragonfruits still needs to improve. Except for the damage of appearance, weight is the main grading index. Dragonfruits can't be graded by its ripeness and taste. Our research use the appearance of dragonfruits to predict the grade of its ripeness by using Convolution Neural Network(CNN) model, which helps to reduce labor and time costs and increase the profit for farmers. In addition, we also develop a real-time ripeness classification system, which combine our research with fruit gravity classifier in the dragonfruit farm and apply our prediction result on existing field.

First, we capture dragonfruits' images by using IP camera for data collection. Then, use object detection methods and data pre-processing to process each image. Finally, we put all images into CNN model for training. Convolution layers extract features from the appearance of dragonfruits, and fully connection layers use these extracted features to classify for predicting ripeness of each dragonfruit. During the prediction stage, we use the identical IP camera to capture image and put it into real-time ripeness classification system for edge computing. Then, transmit the predicted result to fruit gravity classifier for grading. Our model predict ripeness of dragonfruits to three classes. Overall accuracy of our model is 94.1%, and f1-score of each class is over than 90%.

Contents
Abstract . . . . . i
中文摘要 . . . . . iii
Contents . . . . . iv
List of Figures . . . . . viii
List of Tables . . . . . x
Chapter 1 Introduction . . . . . 1
Chapter 2 Background and Related Works . . . . . 5
2.1 Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Learning Concepts . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Convolution Neural Network . . . . . . . . . . . . . . . . . . . . . . 9
2.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.2 Learning Concepts . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 YOLO v3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1 Object Detection . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 YOLO v3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Deep Residual Network . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.1 Residual Learning . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.2 ResNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 3 System Architecture . . . . . 21
3.1 Hardware Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.1.1 Fruit Gravity Classifier . . . . . . . . . . . . . . . . . . . . . 23
3.1.2 IP Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.3 Computing Resource . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.1 Process Frames on OpenCV . . . . . . . . . . . . . . . . . . 26
3.2.2 Object Detection Module . . . . . . . . . . . . . . . . . . . 27
3.3 Real-Time Ripeness Prediction . . . . . . . . . . . . . . . . . . . . 28
3.3.1 Data Pre-Processing . . . . . . . . . . . . . . . . . . . . . . 28
3.3.2 Ripeness prediction module . . . . . . . . . . . . . . . . . . 28
3.3.3 Classification mechanism module . . . . . . . . . . . . . . . 29
Chapter 4 System Implementation 30
4.1 Data Collection Subsystem . . . . . . . . . . . . . . . . . . . . . . . 30
4.1.1 Record Video of Dragonfruits . . . . . . . . . . . . . . . . . 30
4.1.1.1 Shooting Environment . . . . . . . . . . . . . . . . 30
4.1.1.2 Parameter Adjustment of IP Camera . . . . . . . . 32
4.1.1.3 Video Recording . . . . . . . . . . . . . . . . . . . 33
4.1.2 Image Extraction from Video . . . . . . . . . . . . . . . . . 34
4.1.2.1 Labeling on Object Detection . . . . . . . . . . . . 35
4.1.2.2 Training on Object Detection . . . . . . . . . . . . 36
4.1.2.3 Image Extraction with Object Detection . . . . . . 39
4.2 Real-time prediction subsystem . . . . . . . . . . . . . . . . . . . . 42
4.2.1 Process on Data Pre-processing . . . . . . . . . . . . . . . . 42
4.2.1.1 Data Augmentation . . . . . . . . . . . . . . . . . 42
4.2.1.2 Image Resize and Standardization . . . . . . . . . 43
4.2.2 Build up Deep Convolution Neural Network Model . . . . . 44
4.2.2.1 Ripeness Labeling on Dragonfruit . . . . . . . . . 44
4.2.2.2 Model Structure Design . . . . . . . . . . . . . . . 45
4.2.2.3 Training and Hyper-parameter Adjustment . . . . 48
4.2.3 Integrate with Real-Time Classification Mechanism Module 48
4.2.3.1 Prediction on Ripeness Classification Model . . . . 49
4.2.3.2 Transfer Result to Refitted Fruit Gravity Classifier 50
Chapter 5 Experiment and Result 53
5.1 Ripeness Classification on Different Model . . . . . . . . . . . . . . 53
5.1.1 Performance of DRC-7 Model . . . . . . . . . . . . . . . . . 56
5.1.2 Performance of ResNet-Based Model . . . . . . . . . . . . . 57
5.1.3 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2 Ripeness Classification on Different Scales of Dataset . . . . . . . . 60
5.2.1 Half Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.2.2 10 Percent of Data . . . . . . . . . . . . . . . . . . . . . . . 63
5.2.3 1 Percent of Data . . . . . . . . . . . . . . . . . . . . . . . . 64
5.2.4 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.3 Real-Time Prediction on Different Computing Resources . . . . . . 67
5.3.1 Performance on NVIDIA Jetson TX2 . . . . . . . . . . . . . 67
5.3.2 Performance on PC with NVIDIA GeForce GTX 1060 . . . 68
Chapter 6 Conclusion and Future Work . . . . . 69
6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Bibliography . . . . . 72

[1] C. Yu, “Development of Taiwan Pitaya Fruit Industry,”Taichung DistrictAgricultural Research and Extension Station, COA, Taiwan, vol. 131, pp.1–12, 2016.[2] H. Yin, F. Fan, J. Zhang, H. Li, and T. Lau, “The Importance of DomainKnowledge,” 2020.[3] M. T. Islam, B. N. Karim Siddique, S. Rahman, and T. Jabid, “Image recog-nition with deep learning,” in2018 International Conference on IntelligentInformatics and Biomedical Sciences (ICIIBMS), vol. 3, 2018, pp. 106–110.[4] H. Fujiyoshi, T. Hirakawa, and T. Yamashita, “Deep learning-based imagerecognition for autonomous driving,”IATSS Research, vol. 43, no. 4, pp.244–252, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0386111219301566[5] Z. M. Fadlullah, F. Tang, B. Mao, N. Kato, O. Akashi, T. Inoue, and K. Mizu-tani, “State-of-the-art deep learning: Evolving machine intelligence towardtomorrow’s intelligent network traffic control systems,”IEEE Communica-tions Surveys Tutorials, vol. 19, no. 4, pp. 2432–2455, 2017.72
[6] X. Xie, J. Niu, X. Liu, Z. Chen, S. Tang, and S. Yu, “A surveyon incorporating domain knowledge into deep learning for medical imageanalysis,”Medical Image Analysis, vol. 69, p. 101985, Apr 2021. [Online].Available: http://dx.doi.org/10.1016/j.media.2021.101985[7] Y. Lu and S. Young, “A survey of public datasets for computervision tasks in precision agriculture,”Computers and Electronics inAgriculture, vol. 178, p. 105760, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0168169920312709[8] N.-F. Huang, D.-L. Chou, and C.-A. Lee, “Real-time classification of greencoffee beans by using a convolutional neural network,” in2019 3rd In-ternational Conference on Imaging, Signal Processing and Communication(ICISPC), 2019, pp. 107–111.[9] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,”CoRR,vol. abs/1804.02767, 2018. [Online]. Available: http://arxiv.org/abs/1804.02767[10] Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” inNature PublishingGroup, vol. 521, 2015, pp. 436–444.[11] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmannmachines,” inProceedings of the 27th International Conference on MachineLearning, 2010, p. 807–814.[12] R. Pascanu, T. Mikolov, and Y. Bengio, “Understanding the explodinggradient problem,”CoRR, vol. abs/1211.5063, 2012. [Online]. Available:http://arxiv.org/abs/1211.506373
[13] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representationsby back-propagating errors,” inNature Publishing Group, vol. 323, 1986, pp.533–536.[14] R. Hecht-Nielsen, “Theory of the backpropagation neural network,” inIn-ternational 1989 Joint Conference on Neural Networks, 1989, pp. 593–605vol.1.[15] Y. Lecun and Y. Bengio,Convolutional networks for images, speech, andtime-series. MIT Press, 1995.[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classificationwith deep convolutional neural networks,” inAdvances in NeuralInformation Processing Systems, F. Pereira, C. J. C. Burges, L. Bottou,and K. Q. Weinberger, Eds., vol. 25. Curran Associates, Inc.,2012. [Online]. Available: https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf[17] Z. Yang, S. Liu, u. H. H, L. Wang, and S. Lin, “Reppoints: Point set repre-sentation for object detection,” in2019 IEEE/CVF International Conferenceon Computer Vision (ICCV), 2019, pp. 9656–9665.[18] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie,“Feature pyramid networks for object detection,” in2017 IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936–944.[19] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for denseobject detection,” in2017 IEEE International Conference on Computer Vision(ICCV), 2017, pp. 2999–3007.74
[20] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer,Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy, “Speed/accuracy trade-offs for modern convolutional object detectors,” in2017 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2017, pp. 3296–3297.[21] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once:Unified, real-time object detection,” in2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2016, pp. 779–788.[22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in2016 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2016, pp. 770–778.[23] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep networktraining by reducing internal covariate shift,” inProceedings of the32nd International Conference on Machine Learning, ser. Proceedingsof Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37.Lille, France: PMLR, 07–09 Jul 2015, pp. 448–456. [Online]. Available:http://proceedings.mlr.press/v37/ioffe15.html[24] “NVIDIA Jetson Xavier NX,” https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/,accessed:2021-06-25.[25] “Open Source Computer Vision Library,” https://opencv.org, accessed: 2021-06-25.[26] “opencv-python - Python Package(OPENCV),” https://github.com/opencv/opencv-python, accessed: 2021-06-25.75
[27] “ImageNet Database,” https://www.image-net.org/, accessed: 2021-06-25.[28] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in py-torch,” 2017.[29] R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like envi-ronment for machine learning,” inBigLearn, NIPS Workshop, 2011.[30] H. Schulzrinne, A. Rao, and R. Lanphier,RFC2326 - Real TimeStreaming Protocol (RTSP), Std. rfc2326, April 1998. [Online]. Available:http://www.ietf.org/rfc/rfc2326.txt[31] “ONVIF,” https://www.onvif.org/, accessed: 2021-06-25.[32] “Gstreamer,” https://gstreamer.freedesktop.org/, accessed: 2021-06-25.[33] “VLC Media Player,” https://www.videolan.org/, accessed: 2021-06-25.[34] “LabelImg,” https://github.com/tzutalin/labelImg, accessed: 2021-06-26.[35] “YOLO v3 Model Structure,” https://pjreddie.com/darknet/yolo/, accessed:2021-06-26.[36] G. D. Finlayson, B. Schiele, and J. L. Crowley, “Comprehensive colour imagenormalization,” 1998.[37] Y. Wu and K. He, “Group normalization,”CoRR, vol. abs/1803.08494, 2018.[Online]. Available: http://arxiv.org/abs/1803.08494[38] “yolo34py - Python package,” https://github.com/madhawav/YOLO3-4-Py,accessed: 2021-06-27.76
[39] “jetson-gpio - Python package,” https://github.com/NVIDIA/jetson-gpio,accessed: 2021-06-27.[40] “FT232R Datasheet,” http://www.ftdichip.com/Support/Documents/DataSheets/Cables/DS_TTL-232R_CABLES.pdf, accessed: 2021-06-27.[41] “pylibftdi - Python package,” https://pylibftdi.readthedocs.io/en/latest/, ac-cessed: 2021-06-27.[42] “NVIDIA Jetson TX2 Module,” https://developer.nvidia.com/embedded/jetson-tx2, accessed: 2021-06-27