近年來受惠於車聯網(V2X)技術的快速進展以及車輛感測器的佈署，各式蒐集的資料得以與鄰近的運算裝置透過車聯網進行交換，實現了智慧交通系統。然而由於車載網路環境具高度動態變化以及高QoS要求等特性，激勵研究者開始應用新的方法來設計車載網路相關演算法。受惠於車感測器蒐集的大量資料，一種人工智慧的算法，強化式學習(RL)，近年來被大量應用於車聯網領域來分析蒐集的資料進而提升相關性能。而RL的運作形式為算法不斷與模擬環境進行互動進而獲取資訊，並透過獲得的資訊進行決策優化。為進行相關算法的開發與探討，支援允許外部程式不斷互動並影響模擬環境的平台，變成不可或缺。目前有一廣泛被使用的強化式學習API，OpenAI-Gym，提供了大量支援RL運作形式的模擬環境，讓開發者得以進行相關探討。但OpenAI-Gym並未支援車載網路模擬環境，且現存的車載網路模擬器也皆未支援相關的RL API。因此在本篇論文中，我們設計與實作了一RL框架，V2X-Gym。 該框架提供了一整合網路模擬器(ns-3)與道路交通模擬器(SUMO)的車載網路模擬器，此外該模擬器也同時支援OpenAI-Gym API。另外我們還在本篇文章中提出了一透過RL算法，Q-learning，來對3GPP車對車通訊演算法，感知式半持續性排程，進行通訊可靠度的提升。並透過V2X-Gym框架對該算法進行模擬與分析，進而驗證V2X-Gym的可行性與適用性。

Recently, vehicles are equipped with sensors to collect information and communicate with surroundings, which enables the operations of the intelligent transportation system (ITS). The key enabler of these connectivity is vehicle-to-everything (V2X) communications. However, a variety of new challenges are produced due to the highly dynamic vehicular network environment, hence motivate the new methodology of designing vehicular network algorithms. Thanks to the large volumes of information collected by the vehicular sensors, reinforcement learning (RL) is introduced to exploit such collected data for enhancing the vehicular network performance. To evaluate RL algorithms, a simulation environment allowing interactions with agents for learning optimal policy is required. We note that, there is a de-facto RL framework called OpenAI-Gym, which provides lots of simulation environment allowing interactions with agents via standardized methods. However, OpenAI-Gym does not provide vehicular network simulation environment, and none of vehicular network simulators provide related RL APIs to allow evaluating/comparing proposed RL-enhanced vehicular network algorithms. In this paper, we design and implement an RL framework called V2X-Gym which contains a proposed vehicular network simulator implemented via integrating network simulator (ns-3) and traffic simulator (SUMO), and encapsulated behind OpenAI-Gym API to provide standardized environment for evaluating RL-enhanced vehicular network algorithms. Furthermore, we adopt an RL algorithm, Q-learning, to improve the reliability of 3GPP V2V sensing-based semi-persistent scheduling (SPS) algorithm. We evaluate this proposed algorithm via the V2X-Gym to prove the feasibility and applicability of our platform.

摘要-----------------------------------------------i
Abstract------------------------------------------ii
Contents-----------------------------------------iii
List of Figures-----------------------------------vi
List of Tables-----------------------------------vii
1. Introduction------------------------------------1
2. Background--------------------------------------5
3. V2X-Gym Reference Toolkit-----------------------9
4. The Architecture of the V2X-Gym Framework------13
5. The Implementation of the V2X-Gym Framework----20
6. Illustrative Example using V2X-Gym-------------32
7. Conclusion-------------------------------------48

[1] K. Sjoberg, P. Andres, T. Buburuzan, and A. Brakemeier, “Cooperative intelligent transport systems in europe: Current deployment status and outlook,” IEEE Vehicular Technology Magazine, vol. 12, no. 2, pp. 89–97, 2017.
[2] X. Cheng, L. Yang, and X. Shen, “D2d for intelligent transportation systems: A feasibility study,” IEEE Transactions on Intelligent Transportation Systems, vol. 16, no. 4, pp. 1784–1793, 2015.
[3] L. Liang, H. Ye, and G. Y. Li, “Toward intelligent vehicular networks: A machine learning framework,” IEEE Internet of Things Journal, vol. 6, no. 1, pp. 124–135, 2018.
[4] G. Association et al., “An assessment of lte-v2x (pc5) and 802.11 p direct communications technologies for improved road safety in the eu,” 5G Automotive Association, Tech. Rep., 2017.
[5] P. Lison, “An introduction to machine learning,” Language Technology Group (LTG), vol. 1, no. 35, 2015.
[6] Z. Li, C. Wang, and C.-J. Jiang, “User association for load balancing in vehicular networks: An online reinforcement learning approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 8, pp. 2217–2228, 2017.
[7] L. Yao, J. Wang, X. Wang, A. Chen, and Y. Wang, “V2x routing in a vanet based on the hidden markov model,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 3, pp. 889–899, 2017.
[8] F. Zeng, R. Zhang, X. Cheng, and L. Yang, “Channel prediction based scheduling for data dissemination in vanets,” IEEE Communications Letters, vol. 21, no. 6, pp. 1409–1412, 2017.
[9] H. Zhou, X. Wang, Z. Liu, Y. Ji, and S. Yamada, “Resource allocation for svc streaming over cooperative vehicular networks,” IEEE Transactions on Vehicular Technology, vol. 67, no. 9, pp. 7924 7936, 2018.
[10] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016.
[11] M. J. Silva, G. I. Silva, C. M. Ferreira, F. A. Teixeira, and R. A. Oliveira, “Survey of vehicular network simulators: A temporal approach,” in International Conference on Enterprise Information Systems. Springer, 2018, pp. 173–192.
[12] M. Rondinone, J. Maneros, D. Krajzewicz, R. Bauza, P. Cataldi, F. Hrizi, J. Gozalvez, V. Kumar, M. Röckl, L. Lin et al., “itetris: a modular simulation platform for the large scale evaluation of cooperative its applications,” Simulation Modelling Practice and Theory, vol. 34, pp. 99–125, 2013.
[13] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. WieBner, “Microscopic traffic simulation using sumo,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2018, pp. 2575–2582.
[14] G. F. Riley and T. R. Henderson, “The ns-3 network simulator,” in Modeling and tools for network simulation. Springer, 2010, pp. 15–34.
[15] P.-H. Hsieh, “V2x-gym: a reinforcement learning framework for evaluating rl-enhanced vehicular network algorithms,” 2020. [Online]. Available: https: //github.com/a3794110/V2X-Gym
[16] Y. Jeon, S. Kuk, and H. Kim, “Reducing message collisions in sensing-based semipersistent scheduling (sps) by using reselection lookaheads in cellular v2x,” Sensors, vol. 18, no. 12, p. 4388, 2018.
[17] D. Bertsekas, Dynamic programming and optimal control. Athena scientific Belmont, MA, 1995, vol. 1, no. 2.
[18] P. Gawlowicz and A. Zubow, “Ns-3 meets openai gym: The playground for machine learning in networking research,” in Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, 2019, pp. 113–120.
[19] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013.
[20] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi agent, reinforcement learning for autonomous driving,” arXiv preprint arXiv:1610.03295, 2016.
[21] C. Wu, K. Parvate, N. Kheterpal, L. Dickstein, A. Mehta, E. Vinitsky, and A. M. Bayen, “Framework for control and deep reinforcement learning in traffic,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), 2017, pp. 1–8.
[22] W. Y. Wang, J. Li, and X. He, “Deep reinforcement learning for nlp,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2018, pp. 19–21.
[23] J. X. Chen, “The evolution of computing: Alphago,” Computing in Science & Engineering, vol. 18, no. 4, pp. 4–7, 2016.
[24] A. Nandy and M. Biswas, Reinforcement Learning with Keras, TensorFlow, and ChainerRL. Berkeley, CA: Apress, 2018, pp. 129–153. [Online]. Available: https://doi.org/10.1007/978-1-4842-3285-9_5
[25] C. Lochert, A. Barthels, A. Cervantes, M. Mauve, and M. Caliskan, “Multiple simulator interlinking environment for ivc,” in Proceedings of the 2nd ACM international workshop on Vehicular ad hoc networks, 2005, pp. 87–88.
[26] T. Issariyakul and E. Hossain, “Introduction to network simulator 2 (ns2),” in Introduction to network simulator NS2. Springer, 2009, pp. 1–18.
[27] H. Wu, J. Lee, M. Hunter, R. Fujimoto, R. L. Guensler, and J. Ko, “Efficiency of simulated vehicle-to-vehicle message propagation in atlanta, georgia, i-75 corridor,” Transportation research record, vol. 1910, no. 1, pp. 82–89, 2005.
[28] M. Killat, F. Schmidt-Eisenlohr, H. Hartenstein, C. Rössel, P. Vortisch, S. Assenmacher, and F. Busch, “Enabling efficient and accurate large-scale simulations of vanets for vehicular traffic management,” in Proceedings of the fourth ACM international workshop on Vehicular ad hoc networks, 2007, pp. 29–38.
[29] M. Piorkowski, M. Raya, A. L. Lugo, P. Papadimitratos, M. Grossglauser, and J.-P. Hubaux, “Trans: realistic joint traffic and network simulator for vanets,” ACM SIGMOBILE mobile computing and communications review, vol. 12, no. 1, pp. 31– 33, 2008.
[30] Y. Pigné, G. Danoy, and P. Bouvry, “A platform for realistic online vehicular network management,” in 2010 IEEE Globecom Workshops. IEEE, 2010, pp. 595–599.
[31] S. Zemouri, S. Mehar, and S.-M. Senouci, “Hints: A novel approach for realistic simulations of vehicular communications,” in 2012 Global Information Infrastructure and Networking Symposium (GIIS). IEEE, 2012, pp. 1–6.
[32] C. Sommer, D. Eckhoff, A. Brummer, D. S. Buse, F. Hagenauer, S. Joerer, and M. Segata, “Veins: The open source vehicular network simulation framework,” in Recent Advances in Network Simulation. Springer, 2019, pp. 215–252.
[33] A. Varga, “Omnet++,” in Modeling and tools for network simulation. Springer, 2010, pp. 35–59.
[34] B. Schünemann, “V2x simulation runtime infrastructure vsimrti: An assessment tool to design smart traffic management systems,” Computer Networks, vol. 55, no. 14, pp. 3189–3198, 2011.
[35] H. Arbabi and M. C. Weigle, “Highway mobility and vehicular ad-hoc networks in ns-3,” in Proceedings of the 2010 Winter Simulation Conference. IEEE, 2010, pp. 2991–3003.
[36] S.-Y. Wang, C. Chou, Y. Chiu, Y. Tzeng, M. Hsu, Y. Cheng, W. Liu, and T. Ho, “Nctuns 4.0: An integrated simulation platform for vehicular traffic, communication, and network researches,” in 2007 IEEE 66th Vehicular Technology Conference. IEEE, 2007, pp. 2081–2085.
[37] L. Bononi, M. D. Felice, G. D’Angelo, M. Bracuto, and L. Donatiello, “Moves: A framework for parallel and distributed simulation of wireless vehicular ad hoc networks,” Computer Networks, vol. 52, no. 1, pp. 155–179, 2008.
[38] A. R. Khan, S. M. Bilal, and M. Othman, “A performance comparison of open source network simulators for wireless networks,” in 2012 IEEE International Conference on Control System, Computing and Engineering, 2012, pp. 34–38.
[39] X. Zeng, R. Bagrodia, and M. Gerla, “Glomosim: a library for parallel simulation of large-scale wireless networks,” in Proceedings. Twelfth Workshop on Parallel and Distributed Simulation PADS ’98 (Cat. No.98TB100233), 1998, pp. 154–161.
[40] E. Egea-Lopez, J. Vales-Alonso, A. Martinez-Sala, P. Pavon Mario, and J. GarciaHaro, “Simulation scalability issues in wireless sensor networks,” IEEE Communications Magazine, vol. 44, no. 7, pp. 64–73, 2006.
[41] J. Bu, G. Tan, N. Ding, M. Liu, and C. Son, “Implementation and evaluation of wave 1609.4/802.11 p in ns-3,” in Proceedings of the 2014 Workshop on ns-3, 2014, pp. 1–8.
[42] F. Eckermann, M. Kahlert, and C. Wietfeld, “Performance analysis of c-v2x mode 4 communication introducing an open-source c-v2x simulator,” in 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall). IEEE, 2019, pp. 1–5.
[43] S. Choi, J. Song, J. Kim, S. Lim, S. Choi, T. T. Kwon, and S. Bahk, “5g k-simnet: End-to-end performance evaluation of 5g cellular systems,” in 2019 16th IEEE Annual Consumer Communications Networking Conference (CCNC), 2019, pp. 1–6.
[44] A. F. A. F, J. E. Espinosa, and J. Espinosa, “Traci4matlab: enabling the integration of the sumo road traffic simulator and matlab R through a software re-engineering process,” in Modeling Mobility with Open Data. Springer, 2015, pp. 155–170.
[45] A. Wegener, M. Piórkowski, M. Raya, H. Hellbrück, S. Fischer, and J.-P. Hubaux, “Traci: an interface for coupling road traffic and network simulators,” in Proceedings of the 11th communications and networking simulation symposium, 2008, pp. 155– 163.
[46] J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in science & engineering, vol. 9, no. 3, pp. 90–95, 2007.
[47] R. Garabík, “Processing xml text with python and elementtree–a practical experience,” INSIGHT INTO THE SLOVAK AND CZECH CORPUS LINGUISTICS, p. 160, 2006.
[48] P. Hintjens, ZeroMQ: messaging for many applications. " O’Reilly Media, Inc.", 2013.
[49] K. Varda, “Protocol buffers: Google’s data interchange format, Google Open Source Blog, Available at least as early as Jul, vol. 72, 2008.
[50] A. Nabil, k. Kaur, c. Dietrich, and V. Marojevic, “Performance analysis of sensingbased semi-persistent scheduling in c-v2x networks,” in 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall). IEEE, 2018, pp. 1–5.