在大基地台覆蓋範圍內密集部署小基地台的異質性網路(HetNets)被視為一種滿足未來5G無線通訊中爆炸性流量需求的可能解決方案之一。然而，由於不同的基地台覆蓋範圍，大基地台和多個低功率的小基地台的共存可能導致負載不平衡的問題。在此類問題中，某些具有服務質量(QoS)要求的移動用戶將遇到服務質量差和不能滿足服務的狀況。在本文中，我們提出一種基於深度強化學習(Deep Reinforcement Learning)的方法來解決用戶連結問題。所提出的方法將根據基地台的負載與用戶需求來確定每個小基地台，適當的基地台範圍擴展(Cell Range Expansion)偏移，以最大化用戶滿意度。此外，我們提出一種結合DRL和啟發式演算法的方法，以減輕訓練階段的性能下降。模擬結果顯示，所提出的基於DRL的方法比基線方法能獲得更好的用戶滿足率。

Heterogeneous networks (HetNets) with small cells densely deployed within the coverage of a macro cell have been considered as one of the promising solutions to meet the explosive mobile traffic demands in the 5G wireless communications. However, the coexistent of a macro cell and multiple low-power small cells may lead to the problem of load imbalance because of the different cell coverages. In such issues, some mobile users with quality of service (QoS) requirements will experience the bad quality and dissatisfaction. In this thesis, we propose a deep reinforcement learning (DRL) based scheme to address the problem of user association. The proposed approach will determine the appropriate cell range expansion (CRE) offset for each small cell depending on the cell loading and user demands to maximize user satisfaction. Furthermore, we propose a scheme that combines both DRL and a heuristic algorithm to alleviate performance degradation in the training phase. Simulation results show that the proposed DRL-based scheme can achieve better user satisfaction than baseline works.

摘要
目錄
1. Introduction-----------1
2. Related Work-----------5
3. System Model-----------9
4. Proposed Algorithm----15
5. Simulation------------32
6. Conclusion------------45
References---------------46

[1] J. G. Andrews, S. Buzzi, W. Choi, S. V. Hanly, A. Lozano, A. C. Soong, and J. C. Zhang, “What will 5G be,” IEEE Journal on Selected Areas in Communications, vol. 32, no. 6, pp. 1065–1082, Jun. 2014.
[2] M. Agiwal, A. Roy, and N. Saxena, ‘‘Next generation 5G wireless networks: A comprehensive survey,’’ IEEE Communications Surveys & Tutorials, vol. 18, no. 3, pp. 1617–1655, Thirdquarter 2016.
[3] M. A. Albreem, “5G wireless communication systems: Vision and challenges,” International Conference on Computer, Communications, and Control Technology (I4CT), Kuching, Malaysia, Apr. 2015.
[4] A. Damnjanovic, J. Montojo, Y. Wei, T. Ji, T. Luo, M. Vajapeyam, T. Yoo, O. Song, and D. Malladi, ‘‘A survey on 3GPP heterogeneous networks,’’ IEEE Wireless Communications, vol. 18, no. 3, pp. 10–21, Jun. 2011.
[5] A. Ghosh, N. Mangalvedhe, R. Ratasuk, B. Mondal, M. Cudak, E. Visotsky, T. A. Thomas, J. G. Andrews, P. Xia, H. S. Jo, H. S. Dhillon, and T. D. Novlan, ‘Heterogeneous cellular networks: from theory to practice’, IEEE Communications Magazine, vol. 50, no. 6, pp. 54–64, Jun. 2012.
[6] R1-083813, “Range expansion for efficient support of heterogeneous networks,” Qualcomm Europe, Sep. 2008.
[7] K. Okino, T. Nakayama, C. Yamazaki, H. Sato, and Y. Kusano, ‘‘Pico cell range expansion with interference mitigation toward LTE-advanced heterogeneous networks,’’ IEEE International Conference on Communications Workshops (ICC), Kyoto, Japan, Jun. 2011.
[8] T. Kudo and T. Ohtsuki, “Cell range expansion using distributed Q-Learning in heterogeneous networks,” IEEE 78th Vehicular Technology Conference (VTC Fall), Las Vegas, NV, USA, Sep. 2013.
[9] K. Kitagawa, T. Komine, T. Yamamoto, and S. Konishi, “Performance evaluation of handover in LTE-Advanced systems with pico cell range expansion,” IEEE International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), Sydney, NSW, Australia, Sep. 2012.
[10] D. López-Pérez and X. Chu, “Inter-cell interference coordination for expanded region picocells in heterogeneous networks,” Proceedings of 20th International Conference on Computer Communications and Networks (ICCCN), Maui, HI, USA, Aug. 2011.
[11] M. Shirakabe, A. Morimoto, and N. Miki, “Performance evaluation of inter-cell interference coordination and cell range expansion in heterogeneous networks for LTE-advanced downlink,” 8th International Symposium on Wireless Communication Systems, Aachen, Germany, Nov. 2011.
[12] M. Al-Rawi, “A dynamic approach for cell range expansion in interference coordinated LTE-advanced heterogeneous networks,” IEEE International Conference on Communication Systems (ICCS), Singapore, Nov. 2012.
[13] P. Tian, H. Tian, J. Zhu, L. Chen, and X. She, “An adaptive bias configuration strategy for range extension in LTE-advanced heterogeneous networks,” IET International Conference on Communication Technology and Application (ICCTA), Beijing, China, Oct. 2011.
[14] K. Kikuchi and H. Otsuka, “Proposal of adaptive control CRE in heterogeneous networks,” IEEE 23rd International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Sydney, NSW, Australia, Sep. 2012.
[15] S. Nakazawa, N. Naganuma, and H. Otsuka, ‘‘Enhanced adaptive control CRE in heterogeneous networks,’’ 14th IEEE Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA, Jan. 2017.
[16] S. L. Su, T. H. Chih, and S. B. Wu, “A novel handover process for mobility load balancing in LTE heterogeneous networks” IEEE International Conference on Industrial Cyber Physical Systems (ICPS), Taipei, Taiwan, May 2019.
[17] H. P. Kuribayashi, M. A. De Souza, D. D. A. Gomes, K. D. C. Silva, M. S. Da Silva, J. C. W. A. Costa, and C. R. L. Francês, “Particle swarm-based cell range expansion for heterogeneous mobile networks”, IEEE Access, vol. 8, pp. 37021–37034, Feb. 2020.
[18] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT Press Cambridge, 1998.
[19] M. Simsek, M. Bennis, and A. Czylwik, “Dynamic inter-cell interference coordination in HetNets: A reinforcement learning approach,” IEEE Global Communications Conference (Globecom), Anaheim, CA, USA, Dec. 2012.
[20] M. Jaber, M. A. Imran, R. Tafazolli, and A. Tukmanov, “A distributed SON-based user-centric backhaul provisioning scheme,” IEEE Access, vol. 4, pp. 2314–2330, May 2016.
[21] P. V. Klaine, M. Jaber, R. D. Souza, and M. A. Imran, ‘‘Backhaul aware user-specific cell association using Q-Learning,’’ IEEE Transactions on Wireless Communications, vol. 18, no. 7, pp. 3528–3541, Jul. 2019.
[22] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
[23] Y. Cao, L. Zhang, and Y. C. Liang, “Deep reinforcement learning for multi-user access control in UAV networks,” IEEE International Conference on Communications (ICC), Shanghai, China, May 2019.
[24] M. Sana, A. De Domenico, and E. C. Strinati, “Multi-agent deep reinforcement learning based user association for dense mmWave networks,” IEEE Global Communications Conference (Globecom), Waikoloa, HI, USA, Dec. 2019.
[25] N. Zhao, Y. C. Liang, D. Niyato, Y. Pei, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous networks,” IEEE Global Communications Conference (Globecom), Abu Dhabi, United Arab Emirates, Dec. 2018.
[26] N. Zhao, Y. C. Liang, D. Niyato, Y. Pei, M. Wu, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,” IEEE Transactions on Wireless Communications, vol. 18, no. 11, pp. 5141–5152, Nov. 2019.
[27] Z. Wang, L. Li, Y. Xu, H. Tian, and S. Cui, “Handover optimization via asynchronous multi-user deep reinforcement learning,” IEEE International Conference on Communications (ICC), Kansas City, MO, USA, May 2018.
[28] Z. Wang, L. Li, Y. Xu, H. Tian, and S. Cui, “Handover control in wireless systems via asynchronous multiuser deep reinforcement learning,” IEEE Internet of Things Journal, vol. 5, no. 6, pp. 4296–4307, Dec. 2018.
[29] Y. Xu, W. Xu, Z. Wang, J. Lin, and S. Cui, “Load balancing for ultradense networks: A deep reinforcement learning-based approach,” IEEE Internet of Things Journal, vol. 6, no. 6, pp. 9399–9412, Dec. 2019.
[30] Evolved Universal Terrestrial Radio Access Network (E-UTRAN); X2 application protocol, document TS 36.423, v16.1.0, 3rd Generation Partnership Project, Mar. 2020.
[31] Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Frequency System Scenarios, document TR 36.942, v15.0.0, 3rd Generation Partnership Project, Jun. 2018.
[32] V. Noroozi, and A. Abbasfar, “Adaptive load balancing in LTE-Advanced networks with base station coordination,” 16th International Symposium on Wireless Personal Multimedia Communications (WPMC), Atlantic City, NJ, USA, Jun. 2013.
[33] N. Bhushan, J. Li, D. Malladi, R. Gilmore, D. Brenner, A. Damnjanovic, R. T. Sukhavasi, C. Patel, and S. Geirhofer, “Network densification: The dominant theme for wireless evolution into 5G,” IEEE Communications Magazine, vol. 52, no. 2, pp. 82–89, Feb. 2014.
[34] S. Keshav, An engineering approach to computer networking: ATM networks, the internet, and the telephone network. Reading: Addison-Wesley, 1997.
[35] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y. C. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” IEEE Communications Surveys & Tutorials, vol. 21, no. 4, pp. 3133–3174, Fourthquarter 2019.
[36] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay”, arXiv preprint arXiv:1511.05952, 2015.
[37] L. Xiao, Y. Li, C. Dai, H. Dai, and H. V. Poor, “Reinforcement learning-based NOMA power allocation in the presence of smart jamming”, IEEE Transactions on Vehicular Technology, vol. 67, no. 4, pp. 3377–3389, Apr. 2018.
[38] Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control, document TS 36.331, v16.0.0, 3rd Generation Partnership Project, Mar. 2020.
[39] R. Liu and J. Zou, “The effects of memory replay in reinforcement learning,” 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA, Oct. 2018.
[40] B. Galkin, J. Kibilda, and L. A. DaSilva, “Backhaul for low-altitude UAVs in urban environments,” IEEE International Conference on Communications (ICC), Kansas City, MO, USA, May 2018.
[41] A. Sharifian, R. Schoenen, H. Yanikomeroglu, G. Senarath, H. T. Cheng, and P. Djukic, “System and method for network resource allocation considering user experience, satisfaction and operator interest,” U.S. Patent Application No 14/181,160, 2014