本研究以彈性製造系統中的棋盤格狀無人搬運車(Automated Guided Vehicle, AGV)系統為例，所探討的AGV系統議題分成兩大部分：AGV派車與棋盤格狀AGV路徑規劃。在AGV派車規劃部分本研究著重於將機器學習建構於AGV派車問題，分別使用監督式學習方法SVM (Support Vector Machine)與強化學習方法DQN (Deep Q Network)進行AGV派車。
本研究提出以考量AGV路徑之未來壅塞程度之路徑規劃法A Star with Future Congestion，使所規劃出的AGV路徑降低AGV可能碰撞、鎖死之情形。同時也考量棋盤格狀AGV中特有的AGV碰撞情形，以追求棋盤格狀AGV系統中有效率且可行之AGV行走路徑。實驗結果發現本研究所提出的A Star with Future Congestion於各個車數情境中，效果皆優於其他路徑規劃方法。
於AGV派車的部分，本研究提出將SVM派車應用於AGV派車的做法，包含如何挑選特徵值、標籤值、以及如何透過模擬生成訓練樣本。並且將SVM派車法與單一派車法則進行實驗比較，以驗證所建構法方之效能與其可行性，實驗結果顯示SVM派車法之績效優於使用單一派車法則。
除了SVM，本研究也提出了DQN派車的作法，包含狀態、動作、獎勵值之設定。DQN派車法之優勢為當環境變化程度較大時，具有動態適應環境之能力，使派車之績效維持於較佳的狀態，最後於實驗中也驗證DQN派車法具有較佳的適應能力。

The path planning problem and the dispatching problem about AGVs system will be discussed in the research. In AGVs path planning problem, a path planning method “A Star with Future Congestion” is proposed by considering the congestion cost of the nodes to be planned. This research also proposed a dead-lock resolution algorithm to deal with the conflict and dead-lock problem in the operation of multi AGVs system.
In the AGVs dispatching issue, an AGV dynamic dispatching in Flexible Manufacturing System (FMS) by machine learning technique is presented in this paper. The objective is to minimize mean tardiness of orders in FMS. The machine learning-based AGV dispatching approach - support vector machine (SVM) AGV dispatcher is proposed. The idea of the dispatcher is to make dispatching decision base on the system attributes. The simulation runs will be carrying out for generating the training data for SVM. The system attributes that might affect the performance of machine learning dispatcher will also be discussed.
Finally, another machine learning-based approach of AGV dispatching is proposed. The Deep Q Network (DQN) dispatcher, which is able to dynamically adjust the dispatching policy depend on the reward function, will be discussed. The definition of states, actions, and rewards for AGV dispatching problem are main issues for this paper.
In the AGVs dispatching experientment, SVM dispatcher has better effiency when the manufacturing situation is closed to the simulation model for training datas. The experiement result also shows that DQN dispatcher has better adaptivity to environment when the system status changes dramatically from time to time.

摘要-i
Abstract-ii
圖目錄-vi
表目錄-vii
第一章 緒論-1
1.1研究背景與動機-1
1.2研究目的-5
第二章 文獻回顧-7
2.1 機器學習背景知識-7
2.1.1 機器學習-7
2.1.2 Support Vector Machine-8
2.1.3 強化學習-11
2.1.4 Deep Q Network-14
2.2 AGV派車應用相關文獻-17
2.3 機器學習於派車應用相關文獻-18
2.4 AGV路徑規劃相關文獻-21
第三章 AGV路徑規劃問題-25
3.1 問題定義-25
3.2 研究方法-26
3.2.1改良式A star路經規劃法-26
3.2.2 避免碰撞機制-27
3.3實驗情境與設定-31
3.4 結果與分析-34
第四章 SVM於AGV派車問題-36
4.1 問題定義-36
4.2 SVM派車模型-39
4.2.1 設計概念-39
4.2.2 AGVs派車法則-39
4.2.3 AGVs系統屬性-41
4.2.4 訓練樣本生成-45
4.3 實驗情境與實驗設定-47
4.4 結果與分析-50
第五章 強化學習於AGV派車問題-55
5.1 DQN派車模型-55
5.1.1 設計概念-55
5.1.2 AGVs派車法則-56
5.1.3 AGVs系統屬性-57
5.1.4 訓練樣本-58
5.1.5獎勵值設定-60
5.1.6 初始神經網路-62
5.2 實驗情境與設定-62
5.3 結果與分析-66
第六章 結論與建議-79
6.1 結論-79
6.2 建議-81
參考文獻-83

[1] 棋盤格式AGV派車系統，取自
https://www.itri.org.tw/chi/Content/MessagessPIC/contents.aspx?&SiteID=1&MmmID=711042252767750327&CatID=745357762033764061&MSID=745360007153102204
[2] Amami, Rimah, Dorra Ben Ayed, and Noureddine Ellouze. "An empirical comparison of SVM and some supervised learning algorithms for vowel recognition." arXiv preprint arXiv:1507.06021 (2015). Available from:
https://arxiv.org/abs/1507.06021
[3] Arulkumaran, Kai, et al. "A brief survey of deep reinforcement learning." arXiv preprint arXiv:1708.05866 (2017). Availible from:
A brief survey of deep reinforcement learning.
[4] amazonrobotic. Available from:
https://www.amazonrobotics.com/#/
[5] Burges, Christopher JC. "A tutorial on support vector machines for pattern recognition." Data Mining and Knowledge Discovery2.2 (1998): 121-167.
[6] Cleveland, William S., and Susan J. Devlin. "Locally weighted regression: an approach to regression analysis by local fitting." Journal of the American Statistical Association 83.403 (1988): 596-610.
[7] Cortes, Corinna, and Vladimir Vapnik. "Support-vector networks." Machine Learning 20.3 (1995): 273-297.
[8] Chen, Ci, et al. "A reinforcement learning based approach for a multiple-load carrier scheduling problem." Journal of Intelligent Manufacturing 26.6 (2015): 1233-1245.
[9] Choe, Ri, Jeongmin Kim, and Kwang Ryel Ryu. "Online preference learning for adaptive dispatching of AGVs in an automated container terminal." Applied Soft Computing 38 (2016): 647-660.
[10] Deisenroth, Marc Peter, Gerhard Neumann, and Jan Peters. "A survey on policy search for robotics." Foundations and Trends in Robotics 2.1–2 (2013): 1-142.
[11] Egbelu, Pius J., and Jose MA Tanchoco. "Characterization of automatic guided vehicle dispatching rules." International Journal of Production Research 22.3 (1984): 359-374.
[12] Guizzo, Eric. "Three engineers, hundreds of robots, one warehouse." IEEE Spectrum 45.7 (2008): 26-34.
[13] Geek 智慧機器人物流專家網站，取自
http://www.geekplus.com.cn/show?catid=3
[14] Hsu, Chih-Wei, Chih-Chung Chang, and Chih-Jen Lin. "A practical guide to support vector classification." (2003): 1-16. Availible from:
https://www.researchgate.net/profile/Chenghai_Yang/publication/272039161_Evaluating_unsupervised_and_supervised_image_classification_methods_for_mapping_cotton_root_rot/links/55f2c57408ae0960a3897985/Evaluating-unsupervised-and-supervised-image-classification-methods-for-mapping-cotton-root-rot.pdf
[15] Kotsiantis, Sotiris B., I. Zaharakis, and P. Pintelas. "Supervised machine learning: A review of classification techniques." Emerging Artificial Intelligence Applications in Computer Engineering 160 (2007): 3-24.
[16] Lawrence, S. (1984). Resource Constrained Project Scheduling: An Experimental Investigation of Heuristic Scheduling Techniques (Supplement), Technical report, Graduate School of Industrial Administration, Carnegie Mellon University, Pittsburgh, PA.
[17] Li, Jun-tao, and Hong-jian Liu. "Design Optimization of Amazon Robotics." Automation, Control and Intelligent Systems 4.2 (2016): 48-52.
[18] Maxwell, William L., and Jack A. Muckstadt. "Design of automatic guided vehicle systems." IIE Transactions 14.2 (1982): 114-124.
[19] Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013). Availible from:
https://arxiv.org/abs/1312.5602
[20] Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533.
[21] Marsland, Stephen. Machine learning: an algorithmic perspective. CRC press, (2015).
[22] Mousavi, Maryam, et al. "Multi-objective AGV scheduling in an FMS using a hybrid of genetic algorithm and particle swarm optimization." PloS One 12.3 (2017): e0169817.
[23] Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (ICML-10). (2010).
[24] Park, Sang Chan, Narayan Raman, and Michael J. Shaw. "Adaptive scheduling in dynamic flexible manufacturing systems: a dynamic rule selection approach." IEEE Transactions on Robotics and Automation 13.4 (1997): 486-502.
[25] Rummery, Gavin A., and Mahesan Niranjan. On-line Q-learning using connectionist systems. Vol. 37. University of Cambridge, Department of Engineering, (1994).
[26] Sutton, Richard S. "Generalization in reinforcement learning: Successful examples using sparse coarse coding." Advances in Neural Information Processing Systems. (1996).
[27] Strehl, Alexander L., et al. "PAC model-free reinforcement learning." Proceedings of the 23rd international conference on Machine learning. ACM, (2006).
[28] Shiue, Yeou-Ren. "Data-mining-based dynamic dispatching rule selection mechanism for shop floor control systems using a support vector machine approach." International Journal of Production Research 47.13 (2009): 3669-3690.
[29] Shiue, Yeou-Ren, Ruey‐Shiang Guh, and Ken‐Chun Lee. "Development of machine learning‐based real time scheduling systems: using ensemble based on wrapper feature selection approach." International Journal of Production Research 50.20 (2012): 5887-5905.
[30] Saidi-Mehrabad, Mohammad, et al. "An Ant Colony Algorithm (ACA) for solving the new integrated model of job shop scheduling and conflict-free routing of AGVs." Computers & Industrial Engineering 86 (2015): 2-13.
[31] Tapan, P. Bagchi. Multiobjective scheduling by genetic algorithms. Springer Science & Business Media, 1999.
[32] Tieleman, Tijmen, and Geoffrey Hinton. "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude." Availible from:
https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
[33] Vis, Iris FA. "Survey of research in the design and control of automated guided vehicle systems." European Journal of Operational Research 170.3 (2006): 677-709.
[34] Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine Learning 8.3-4 (1992): 279-292.
[35] Weston, Jason, and Chris Watkins. Multi-class support vector machines. Technical Report CSD-TR-98-04, Department of Computer Science, Royal Holloway, University of London, May, (1998).
[36] Yuan, Ruiping, Tingting Dong, and Juntao Li. "Research on the Collision-Free Path Planning of Multi-AGVs System Based on Improved A* Algorithm." American Journal of Operations Research 6.06 (2016): 442.
[37] Zeng, Jianyang, and Wen-Jing Hsu. "Off-line AGV routing on the 2D mesh topology with partial permutation." Intelligent Transportation Systems, 2003. Proceedings. 2003 IEEE. Vol. 2. IEEE, (2003).
[38] Zeng, Qingcheng, Zhongzhen Yang, and Xiangpei Hu. "A method integrating simulation and reinforcement learning for operation scheduling in container terminals." Transport 26.4 (2011): 383-393.
[39] Zamiri Marvizadeh, S., and F. F. Choobineh. "Entropy-based dispatching for automatic guided vehicles." International Journal of Production Research 52.11 (2014): 3303-3316.