因資源產能限制而造成機台負載的轉移及不平衡而導致的機器瓶頸，皆削弱了產品混合彈性生產系統的生產積效，因此即時排程控制系統的知識庫應該可以是動態的，並且需包括監控生產系統中發生關鍵變更時的知識修訂機制。本研究提出基於增強式學習即時排程用於支援彈性製造系統和半導體晶圓製造系統的多派工法則選擇機制，所提出之基於增強式學習的即時排程方法包含多派工法則知識庫的建立及修訂的兩個階段。實驗結果顯示，本研究提出之方法產生的系統性能優於採取固定式派工法則、機器學習分類方法和傳統的多派工法則選擇機制。

Machine bottlenecks, resulting from shifting and unbalanced machine loads caused by resource capacity limitations, impair product-mix flexibility production systems. Thus, the knowledge base (KB) of real-time scheduling (RTS) control system should be dynamic and include a knowledge revision mechanism for monitoring crucial changes that occur in the production system. In this research, reinforcement learning (RL)-based RTS and a selection mechanism for multiple dispatching rules (MDRs) are proposed to support the operating characteristics of a flexible manufacturing system (FMS) and semiconductor wafer fabrication (FAB). The proposed RL-based RTS MDRs selection mechanism consisted of initial MDRs KB generation and revision phases. According to various performance criteria, the presented approach yielded a system performance that was superior to those of the fixed-decision scheduling approach, the machine learning classification approach, and the classical MDRs selection mechanism.

摘要 i
ABSTRACT ii
CONTENTS iv
TABLES vi
FIGURES viii
CHAPTER 1 INTRODUCTION 1
1.1 Overview 1
1.2 Research motivations 2
1.3 Research Objectives 3
1.4 Organization 4
CHAPTER 2 THEORETICAL BACKGROUND 5
2.1 Real-time scheduling (RTS) 5
2.2 SOM neural networks 9
2.3 Intelligent Agent 13
2.4 Reinforcement learning (RL) and Q-learning 14
CHAPTER 3 FORMULATION OF THE PROBLEM 18
3.1 Real-time scheduling (RTS) using the MDRs mechanism 18
3.2 Description of a study case 19
3.3 Specification of the training examples 26
CHAPTER 4 DEVELOPMENT OF RL-BASED RTS USING THE MDRS MECHANISM 29
4.1 Simulation-based training example generation mechanism 31
4.2 Data preprocessing mechanism 33
4.3 System state number determination 35
4.4 Initialization of the MDRs KB 36
4.5 Procedure of the MDRs KB revision phase: Q-learning-based agent 38
CHAPTER 5 EXPERIMENT 44
5.1 Simulation model construction and generation of a training example 44
5.2 Feature selection and system state determination in MDRs KB initialization 47
5.3 Simulation experiment verification 48
CHAPTER 6 CONCLUSION AND FUTURE WORK 54
REFERENCES 56

[1] M. Hermann, T. Pentek, B. Otto, Design principles for industrie 4.0 scenarios, in: 2016 49th Hawaii international conference on system sciences (HICSS), IEEE, 2016, pp. 3928-3937.
[2] S. Wang, J. Wan, D. Li, C. Zhang, Implementing smart factory of industrie 4.0: an outlook, International Journal of Distributed Sensor Networks, 12 (2016) 3159805.
[3] J. Lee, B. Bagheri, H.-A. Kao, A cyber-physical systems architecture for industry 4.0-based manufacturing systems, Manufacturing letters, 3 (2015) 18-23.
[4] E.A. Lee, Cyber physical systems: Design challenges, in: 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC), IEEE, 2008, pp. 363-369.
[5] W. Wolf, Cyber-physical systems, Computer, (2009) 88-89.
[6] J. Lee, H.-A. Kao, S. Yang, Service innovation and smart analytics for industry 4.0 and big data environment, Procedia Cirp, 16 (2014) 3-8.
[7] J. Lee, E. Lapira, B. Bagheri, H.-a. Kao, Recent advances and trends in predictive manufacturing systems in big data environment, Manufacturing letters, 1 (2013) 38-41.
[8] A. Goryachev, S. Kozhevnikov, E. Kolbova, O. Kuznetsov, E. Simonova, P. Skobelev, A. Tsarev, Y. Shepilov, “Smart Factory”: Intelligent System for Workshop Resource Allocation, Scheduling, Optimization and Controlling in Real Time, in: Advanced Materials Research, Trans Tech Publ, 2013, pp. 508-513.
[9] M. Kück, J. Ehm, M. Freitag, E.M. Frazzon, R. Pimentel, A data-driven simulation-based optimisation approach for adaptive scheduling and control of dynamic manufacturing systems, in: Advanced Materials Research, Trans Tech Publ, 2016, pp. 449-456.
[10] J. Bengtsson, J. Olhager, The impact of the product mix on the value of flexibility, Omega-Int J Manage S, 30 (2002) 265-273.
[11] M.V. Martin, K. Ishii, Design for variety: developing standardized and modularized product platform architectures, Res Eng Des, 13 (2002) 213-235.
[12] T. Jamrus, C.-F. Chien, M. Gen, K. Sethanan, Hybrid particle swarm optimization combined with genetic operators for flexible job-shop scheduling under uncertain processing time for semiconductor manufacturing, IEEE Trans. Semicond. Manuf., 31 (2017) 32-41.
[13] T. Meng, Q.-K. Pan, H.-Y. Sang, A hybrid artificial bee colony algorithm for a flexible job shop scheduling problem with overlapping in operations, Int J Prod Res, 56 (2018) 5278-5292.
[14] K. Gao, Z. Cao, L. Zhang, Z. Chen, Y. Han, Q. Pan, A review on swarm intelligence and evolutionary algorithms for solving flexible job shop scheduling problems, IEEE/CAA Journal of Automatica Sinica, 6 (2019) 904-916.
[15] A. Shahzad, N. Mebarki, Data mining based job dispatching using hybrid simulation-optimization approach for shop scheduling problem, Eng Appl Artif Intell, 25 (2012) 1173-1181.
[16] B. Saenz de Ugarte, A. Artiba, R. Pellerin, Manufacturing execution system – a literature review, Prod Plan Control, 20 (2009) 525-539.
[17] R.Y. Zhong, Q.Y. Dai, T. Qu, G.J. Hu, G.Q. Huang, RFID-enabled real-time manufacturing execution system for mass-customization production, Robot. Comput. Integr. Manuf., 29 (2013) 283-292.
[18] Y.F. Zhang, W.B. Wang, N.Q. Wu, C. Qian, IoT-Enabled Real-Time Production Performance Analysis and Exception Diagnosis Model, IEEE Trans. Autom. Sci. Eng., 13 (2016) 1318-1332.
[19] J. Leng, P. Jiang, Dynamic scheduling in RFID-driven discrete manufacturing system by using multi-layer network metrics as heuristic information, J Intell Manuf, 30 (2019) 979-994.
[20] D.A. Rossit, F. Tohmé, M. Frutos, Industry 4.0: smart scheduling, Int J Prod Res, 57 (2019) 3802-3813.
[21] C. Qian, Y. Zhang, C. Jiang, S. Pan, Y. Rong, A real-time data-driven collaborative mechanism in fixed-position assembly systems for smart manufacturing, Robot. Comput. Integr. Manuf., 61 (2020) 101841.
[22] C.O. Kim, H.S. Min, Y. Yih, Integration of inductive learning and neural networks for multi-objective FMS scheduling, Int J Prod Res, 36 (1998) 2497-2509.
[23] Y.J. Son, H. Rodriguez-Rivera, R.A. Wysk, A multi-pass simulation-based, real-time scheduling and shop floor control system, T Soc Comput Simul I, 16 (1999) 159-172.
[24] Y.R. Shiue, K.C. Lee, C.T. Su, Real-time scheduling for a smart factory using a reinforcement learning approach, Comput Ind Eng, 125 (2018) 604-614.
[25] P. Priore, A. Gomez, R. Pino, R. Rosillo, Dynamic scheduling of manufacturing systems using machine learning: An updated review, AIEDAM, 28 (2014) 83-97.
[26] J. Mohan, K. Lanka, A.N. Rao, A review of dynamic job shop scheduling techniques, Procedia Manufacturing, 30 (2019) 34-39.
[27] S.-Y.D. Wu, R.A. Wysk, An application of discrete-event simulation to on-line control and scheduling in flexible manufacturing, Int J Prod Res, 27 (2007) 1603-1623.
[28] N. Ishii, J.J. Talavage, A Transient-Based Real-Time Scheduling Algorithm in Fms, Int J Prod Res, 29 (1991) 2501-2520.
[29] Y.R. Shiue, Data-mining-based dynamic dispatching rule selection mechanism for shop floor control systems using a support vector machine approach, Int J Prod Res, 47 (2009) 3669-3690.
[30] G. Metan, I. Sabuncuoglu, H. Pierreval, Real time selection of scheduling rules and knowledge extraction via dynamically controlled data mining, Int J Prod Res, 48 (2010) 6909-6938.
[31] Y.R. Shiue, R.S. Guh, K.C. Lee, Development of machine learning-based real time scheduling systems: using ensemble based on wrapper feature selection approach, Int J Prod Res, 50 (2012) 5887-5905.
[32] J.P.U. Cadavid, S. Lamouri, B. Grabot, R. Pellerin, A. Fortin, Machine learning applied in production planning and control: a state-of-the-art in the era of industry 4.0, J Intell Manuf, (2020) 1-28.
[33] D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning internal representations by error propagation, in, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[34] J.R. Quinlan, C4. 5: programs for machine learning, Elsevier, 2014.
[35] V. Vapnik, The nature of statistical learning theory, Springer science & business media, 2013.
[36] L. Li, Z. Sun, M. Zhou, F. Qiao, Adaptive dispatching rule for semiconductor wafer fabrication facility, IEEE Trans. Autom. Sci. Eng., 10 (2012) 354-364.
[37] Y. Ma, F. Qiao, F. Zhao, J.W. Sutherland, Dynamic scheduling of a semiconductor production line based on a composite rule set, Applied Sciences, 7 (2017) 1052.
[38] P. Priore, B. Ponte, J. Puente, A. Gómez, Learning-based scheduling of flexible manufacturing systems using ensemble methods, Comput Ind Eng, 126 (2018) 282-291.
[39] F. Qiao, Y. Ma, M. Zhou, Q. Wu, A novel rescheduling method for dynamic semiconductor manufacturing systems, IEEE Transactions on Systems, Man, and Cybernetics: Systems, (2018).
[40] N. Ishii, J.J. Talavage, A mixed dispatching rule approach in FMS scheduling, Int J Flex Manuf Syst, 6 (1994) 69-87.
[41] S.-H. Chung, C.-Y. Huang, The Design of Rapid Production Planning Mechanism for the Product Mix Changing in a Wafer Fabrication, J Chin Inst Ind Eng, 20 (2003) 169-176.
[42] C.F. Chien, C.Y. Hsu, C.W. Hsiao, Manufacturing intelligence to forecast and reduce semiconductor cycle time, J Intell Manuf, 23 (2012) 2281-2294.
[43] J. Shahrabi, M.A. Adibi, M. Mahootchi, A reinforcement learning approach to parameter estimation in dynamic job shop scheduling, Comput Ind Eng, 110 (2017) 75-82.
[44] C.J.C.H. Watkins, P. Dayan, Q-Learning, Mach Learn, 8 (1992) 279-292.
[45] Y.C. Wang, J.M. Usher, Application of reinforcement learning for agent-based production scheduling, Eng Appl Artif Intell, 18 (2005) 73-82.
[46] N. Stricker, A. Kuhnle, R. Sturm, S. Friess, Reinforcement learning for adaptive order dispatching in the semiconductor industry, CIRP Ann Manuf Technol, 67 (2018) 511-514.
[47] A. Kuhnle, N. Röhrig, G. Lanza, Autonomous order dispatching in the semiconductor industry using reinforcement learning, Procedia CIRP, 79 (2019) 391-396.
[48] T. Kohonen, Self-Organizing Maps, Springer-Verlag, 2001.
[49] T. Kohonen, The'neural'phonetic typewriter, computer, 21 (1988) 11-22.
[50] Y.-C. Liu, M. Liu, X.-L. Wang, Application of self-organizing maps in text clustering: a review, chapter, 2012.
[51] V.G. Maltarollo, K.M. Honório, A.B.F. da Silva, Applications of artificial neural networks in chemical problems, Artificial neural networks-architectures and applications, (2013) 203-223.
[52] A. Aslantas, D. Emre, M. Çakiroğlu, Comparison of segmentation algorithms for detection of hotspots in bone scintigraphy images and effects on CAD systems, Biomedical Research, 28 (2017) 676-683.
[53] L. Grajciarova, J. Mares, P. Dvorak, A. Prochazka, Biomedical image analysis using self-organizing maps, in: Matlab Conference, 2012.
[54] J.Z. Bloom, Market segmentation: A neural network application, Annals of Tourism Research, 32 (2005) 93-111.
[55] J. Vesanto, E. Alhoniemi, Clustering of the self-organizing map, IEEE Trans. Neural Netw., 11 (2000) 586-600.
[56] D.L. Davies, D.W. Bouldin, A cluster separation measure, IEEE Trans. Pattern Anal. Mach. Intell., 1 (1979) 224-227.
[57] G. Weiss, Multiagent systems: a modern approach to distributed artificial intelligence, MIT press, 1999.
[58] W. Shen, D.H. Norrie, J.-P. Barthès, Multi-agent systems for concurrent intelligent design and manufacturing, CRC press, 2003.
[59] M. Wooldridge, N.R. Jennings, Intelligent agents: Theory and practice, The knowledge engineering review, 10 (1995) 115-152.
[60] R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, 1998.
[61] B. Jang, M. Kim, G. Harerimana, J.W. Kim, Q-learning algorithms: A comprehensive classification and applications, IEEE Access, 7 (2019) 133653-133667.
[62] F. Zhang, J. Leitner, M. Milford, B. Upcroft, P. Corke, Towards vision-based deep reinforcement learning for robotic motion control, arXiv preprint arXiv:1511.03791, (2015).
[63] L. Jiang, H. Huang, Z. Ding, Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge, IEEE/CAA Journal of Automatica Sinica, (2019).
[64] J. Si, Y.-T. Wang, Online learning control by association and reinforcement, IEEE Trans. Neural Netw., 12 (2001) 264-276.
[65] B. Kiumarsi, K.G. Vamvoudakis, H. Modares, F.L. Lewis, Optimal and autonomous control using reinforcement learning: A survey, IEEE Transactions on Neural Networks and Learning Systems, 29 (2017) 2042-2062.
[66] A. Kuhnle, L. Schäfer, N. Stricker, G. Lanza, Design, Implementation and Evaluation of Reinforcement Learning for an Adaptive Order Dispatching in Job Shop Manufacturing Systems, Procedia CIRP, 81 (2019) 234-239.
[67] J.H. Zhong, Z.P. Peng, Q.R. Li, J.G. He, Multi Workflow Fair Scheduling Scheme Research Based on Reinforcement Learning, Procedia Computer Science, 154 (2019) 117-123.
[68] H.-N. Dai, H. Wang, G. Xu, J. Wan, M. Imran, Big data analytics for manufacturing internet of things: opportunities, challenges and enabling technologies, Enterprise Information Systems, (2019) 1-25.
[69] H. Cho, R.A. Wysk, A Robust Adaptive Scheduler for an Intelligent Workstation Controller, Int J Prod Res, 31 (1993) 771-789.
[70] C.-C. Lee, J. Lin, Deadlock prediction and avoidance based on Petri nets for zone-control automated guided vehicle systems, Int J Prod Res, 33 (1995) 3249-3265.
[71] N. Wu, W. Zeng, Deadlock avoidance in an automated guidance vehicle system using a coloured Petri net model, Int J Prod Res, 40 (2002) 223-238.
[72] K. Xing, L. Han, M. Zhou, F. Wang, Deadlock-free genetic scheduling algorithm for automated manufacturing systems based on deadlock control policy, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 42 (2011) 603-615.
[73] M. Montazeri, L.N. Vanwassenhove, Analysis of Scheduling Rules for an FMS, Int J Prod Res, 28 (1990) 785-802.
[74] E. Campbell, J. Ammenheuser, 300 mm factory layout and material handling modeling: Phase II report, in: Tech transfer document, 2000.
[75] L.M. Wein, Scheduling Semiconductor Wafer Fabrication, IEEE Trans. Semicond. Manuf., 1 (1988) 115-130.
[76] S.C.H. Lu, D. Ramaswamy, P.R. Kumar, Efficient Scheduling Policies to Reduce Mean and Variance of Cycle-Time in Semiconductor Manufacturing Plants, IEEE Trans. Semicond. Manuf., 7 (1994) 374-388.
[77] B.W. Hsieh, C.H. Chen, S.C. Chang, Scheduling semiconductor wafer fabrication by using ordinal optimization-based simulation, IEEE Trans. Robot. Autom., 17 (2001) 599-608.
[78] S.C. Park, N. Raman, M.J. Shaw, Adaptive scheduling in dynamic flexible manufacturing systems: A dynamic rule selection approach, IEEE Trans. Robot. Autom., 13 (1997) 486-502.
[79] C.C. Chen, Y. Yih, Y.C. Wu, Auto-bias selection for developing learning-based scheduling systems, Int J Prod Res, 37 (1999) 1987-2002.
[80] Y. Arzi, L. Iaroslavitz, Operating an FMC by a decision-tree-based adaptive production control system, Int J Prod Res, 38 (2000) 675-697.
[81] C.T. Su, Y.R. Shiue, Intelligent scheduling controller for shop floor control systems: a hybrid genetic algorithm/decision tree learning approach, Int J Prod Res, 41 (2003) 2619-2641.
[82] H. Liu, H. Motoda, Feature selection for knowledge discovery and data mining, Springer Science & Business Media, 2012.
[83] H. Liu, R. Setiono, A probabilistic approach to feature selection-a filter solution, in: ICML, 1996, pp. 319-327.
[84] T.P. Simulation, Tecnomatix plant simulation 9.0 user guide, in, Tecnomatix Technologies Ltd Plano, 2009.
[85] P.S. Mahajan, R.G. Ingalls, Evaluation of methods used to detect warm-up period in steady state simulation, in: Proceedings of the 2004 Winter Simulation Conference, 2004., IEEE, 2004.
[86] T. MathWorks, MATLAB Release 2016b, The MathWorks, in: Inc., Natick, Massachusetts, United States, 2016.
[87] H. Demuth, M. Beale, M. Hagan, MATLAB Neural Network Toolbox, Version 5, User’s Guide, in: Natick, Massachusetts N, 2006.
[88] D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley Publishing Company, 1989.
[89] R.S. Guh, Y.R. Shiue, T.Y. Tseng, The study of real time scheduling by an intelligent multi-controller approach, Int J Prod Res, 49 (2011) 2977-2997.
[90] Y.R. Shiue, R.S. Guh, K.C. Lee, Study of SOM-based intelligent multi-controller for real-time scheduling, Appl. Soft Comput., 11 (2011) 4569-4580.