准入控制（admission control）是一個被廣泛討論的問題，尤其在通訊領域中特別重要。通訊系統必須透過准入控制器（admission controller）在使用者開始使用通訊服務之前，先檢視系統以及使用者狀態，之後才做出是否建立該通訊連線的決定。

准入控制器的設計會希望盡可能最大化系統的使用率，但是同時也要能夠兼顧系統中使用者的服務品質（quality of service, QoS），因此如何適量的接收使用者成為一個相當重要的問題。

傳統數據分析的方式已無法有效的應用在多變的情境當中，透過數據分析與閥值（threshold）的設定來判斷接收與否的方式很主觀。因此我們將藉由機器學習（machine learning, ML）的方式，訓練一個准入控制器，使它能夠兼顧使用者的服務品質與系統的使用率。

本文考慮到的使用者屬於5G網路最主要的三種使用類型，分別是增強型行動寬頻類型（Enhanced Mobile Broadband, eMBB）、超可靠度和低延遲類型（Ultra-reliable and Low Latency Communications, URLLC）、大規模機器型類型（Massive Machine Type Communications, mMTC）。在這樣的情境下，我們提出了一個基於深度強化學習（deep reinforcement learning, DRL）的方式來訓練准入控制器，最後將換手（handoff）的因素也列入考慮。

Admission control is an important problem that has been widely discussed for a long time, especially in the network domain. The admission controller needs to check the status of the system and the incoming user before establishing the connection. The goal of the admission controller is to maximize the system utilization while meeting the quality of service constraints of all the users in the system. Due to the complexity of the modern network, machine learning (ML) becomes a feasible solution to deal with the admission control problem, since ML approaches can model the network behavior using the process of learning. In this thesis, the proposed admission control scheme is based on deep reinforcement learning (DRL) and we consider an environment with three main 5G traffic classes defined by ITU-R, including enhanced mobile broadband, ultra-reliable and low latency communications, and massive machine type communications. The proposed DRL-based approach is also capable of handling the admission control problem with handoff users.

中文摘要 ------------------------------------------------------- i
Abstract ------------------------------------------------------ ii
Acknowledgements ---------------------------------------------- iii
List of Figures ----------------------------------------------- vi
List of Tables ------------------------------------------------ viii

1 Introduction ------------------------------------------------ p.1
2 Related work ------------------------------------------------ p.5
3 System Model and Problem Setting ---------------------------- p.8
4 Proposed Scheme --------------------------------------------- p.12
4.1 Background on Reinforcement Learning ---------------------- p.12
4.2 Admission Control using RL -------------------------------- p.16
4.2.1 State, Action, Reward ----------------------------------- p.16
4.2.2 Implementation Challenges ------------------------------- p.19
4.2.3 Prioritized Experience Replay --------------------------- p.24
5 Simulation -------------------------------------------------- p.25
5.1 Experiment Setup ------------------------------------------ p.25
5.2 Simulation Results ---------------------------------------- p.28
5.2.1 Comparison between different objectives ----------------- p.29
5.2.2 The effect of using PER scheme -------------------------- p.31
5.2.3 Compare the proposed scheme with Q-learning and greedy method --------------------------------------------------------------- p.32
5.2.4 Prioritize the handoff users using hyperparameter k ----- p.34
6 Conclusions ------------------------------------------------ p.37

Bibliography -------------------------------------------------- p.37

[1] M. Agiwal, A. Roy, and N. Saxena, “Next Generation 5G Wire- less Networks: A Comprehensive Survey,” IEEE Communications Surveys & Tutorials, vol. 18, no. 3, pp. 1617–1655, 2016.

[2] M. Series, “IMT Vision–Framework and Overall Objectives of the Future Development of IMT for 2020 and Beyond,” Recommendation ITU, vol. 2083, 2015.

[3] A. Parekh and R. Gallager, “A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case,” IEEE Trans. on Networking, vol. 1, no. 3, pp. 344–357, 1993.

[4] A. Parekh and R. Gallager, “A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Multiple Node Case,” IEEE Trans. on Networking, vol. 2, no. 2, pp. 137–150, 1994.

[5] G. Kesidis, J. Walrand, and C. Chang, “Effective Bandwidths for Multiclass Markov Fluids and Other ATM Sources,” IEEE/ACM Trans. on Networking, vol. 1, pp. 424–428, 1993.

[6] A. Elwalid and D. Mitra, “Effective Bandwidth of General Marko- vian Traffic Sources and Admission Control of High-speed Net- works,” ACM/IEEE Trans. on Networking, vol. 1, no. 3, pp. 329– 343, June 1993.

[7] E. Gelenbe, X. Mang, and R. Onvural, “Bandwidth Allocation and Call Admission Control in High-speed Networks,” IEEE Communications Magazine, vol. 35, no. 5, pp. 122–129, 1997.

[8] H. Perros and K. Elsayed, “Call Admission Control Schemes: A Review,” Communications Magazine, IEEE, vol. 34, pp. 82 – 91, 12 1996.

[9] H. G. Perros and K. M. Elsayed, “Call Admission Control Schemes: A Review,” IEEE Communications Magazine, vol. 34, no. 11, pp. 82–91, 1996.

[10] Y. Fang and Y. Zhang, “Call Admission Control Schemes and Performance Analysis in Wireless Mobile Networks,” IEEE Transactions on vehicular technology, vol. 51, no. 2, pp. 371–382, 2002.

[11] R. Boutaba, M. A. Salahuddin, N. Limam, S. Ayoubi, N. Shahriar,
F. Estrada-Solano, and O. M. Caicedo, “A Comprehensive Survey on Machine Learning for Networking: Evolution, Applications and Research Opportunities,” Journal of Internet Services and Applications, vol. 9, no. 1, pp. 1–99, 2018.

[12] T. Sigwele, P. Pillai, A. S. Alam, and Y. F. Hu, “Fuzzy Logic-based Call Admission Control in 5G Cloud Radio Access Networks with Preemption,” EURASIP Journal on Wireless Communications and Networking, vol. 2017, no. 1, pp. 1–10, 2017.

[13] M. Al-Maitah, O. O. Semenova, A. O. Semenov, P. I. Kulakov, and
V. Y. Kucheruk, “A Hybrid Approach to Call Admission Control in 5G Networks,” Advances in Fuzzy Systems, vol. 2018, 2018.

[14] A. Bashar, G. Parr, S. McClean, B. Scotney, and D. Nauck, “Ma- chine Learning-based Call Admission Control Approaches: A Comparative Study,” in 2010 International Conference on Network and Service Management. IEEE, 2010, pp. 431–434.

[15] N. Baldo, P. Dini, and J. Nin-Guerrero, “User-driven Call Admission Control for VoIP over WLAN with a Neural Network-based Cognitive Engine,” in 2010 2nd International Workshop on Cognitive Information Processing. IEEE, 2010, pp. 52–56.

[16] B. Bojovic, N. Baldo, and P. Dini, “A Cognitive Scheme for Radio Admission Control in LTE Systems,” in 2012 3rd International Workshop on Cognitive Information Processing (CIP). IEEE, 2012, pp. 1–3.

[17] T. X. Brown, H. Tong, and S. Singh, “Optimizing Admission Control while Ensuring Quality of Service in Multimedia Networks via Reinforcement Learning,” Advances in Neural Information Processing Systems, pp. 982–988, 1999.

[18] J. Wang and Y. Qiu, “A New Call Admission Control Strategy for LTE Femtocell Networks,” in 2nd International Conference on Advances in Computer Science and Engineering (CSE 2013). Atlantis Press, 2013.

[19] M. Raeis, A. Tizghadam, and A. Leon-Garcia, “Reinforcement Learning-based Admission Control in Delay-Sensitive Service Systems,” in GLOBECOM 2020-2020 IEEE Global Communications Conference. IEEE, 2020, pp. 1–6.

[20] P. Gawłowicz and A. Zubow, “NS-3 Meets OpenAI Gym: The Play- ground for Machine Learning in Networking Research,” in Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, 2019, pp. 113–120.

[21] G. Series, “IMT Vision–End-to-end Quality of Service for Video Telephony over 4G Mobile Networks,” Recommendation ITU, vol. 1028, 2019.

[22] L. Cominardi, L. M. Contreras, C. J. Bcrnardos, and I. Berberana, “Understanding QoS Applicability in 5G Transport Networks,” in 2018 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB). IEEE, 2018, pp. 1–5.

[23] A. Sivanathan, D. Sherratt, H. H. Gharakheili, A. Radford, C. Wi- jenayake, A. Vishwanath, and V. Sivaraman, “Characterizing and Classifying IoT Traffic in Smart Cities and Campuses,” in 2017 IEEE Conference on Computer Communications Workshops (IN-FOCOM WKSHPS). IEEE, 2017, pp. 559–564.

[24] M. Beshley, N. Kryvinska, M. Seliuchenko, H. Beshley, E. M. Shakshuki, A.-U.-H. Yasar et al., “End-to-End QoS “Smart Queue” Management Algorithms and Traffic Prioritization Mechanisms for Narrow-band Internet of Things Services in 4G/5G Networks,” Sensors, vol. 20, no. 8, p. 2324, 2020.

[25] S.-M. Senouci, A.-L. Beylot, and G. Pujolle, “A Dynamic Q- Learning-based Call Admission Control for Multimedia Cellular Networks,” in Proceedings of the 3rd IEEE International Conference in Mobile and Wireless Communication Networks, MWCN2001, Recife, Brazil, 2001, pp. 37–43.