在蜂巢網路中，裝置對裝置通訊（D2D）被視為提高頻譜效率和緩解移動流量爆炸的可能解決方案之一。然而，使用裝置對裝置通訊會對蜂巢網路帶來干擾的問題，進而影響網路整體效能。 在本文中，我們提出了一種基於深度強化式學習 (Deep Reinforcement Learning) 的方法來解決干擾管理和資源分配的問題。 所提出的方法，將根據頻道狀態為每個蜂窩用戶(CU）和D2D發射端（D2D TX）分配適當的復用頻道和傳輸功率，以最大化系統吞吐量。 此外，我們透過結合DRL和啟發式演算法，來提升訓練階段的性能。 模擬結果顯示，所提出基於DRL的集中式方法在系統吞吐量方面優於基線方法。

Device-to-device (D2D) communication is a promising solution to improve spectrum efficiency and alleviate mobile traffic explosion. However, interference mitigation and resource allocation in the underlying cellular network is a tedious and challenging task. This thesis proposes a deep reinforcement learning (DRL) based scheme to solve the interference mitigation and resource allocation problem. According to the channel status, the proposed method will determine the appropriate reuse channel and transmission power for each cellular user (CU) and D2D transmitter (D2D TX) to maximize the system throughput. We combine the DRL scheme and a heuristic algorithm to increase the performance in the training phase. The simulation results show that the proposed algorithm is better than the candidate algorithms in terms of total transmission throughput.

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 System Model and Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . .. . 9
4 Channel Assignment And Power Allocation Algorithm (CAPA) . . . . . . . . 13
4.1 Preliminaries on Reinforcement Learning . . . . . . . . . . . . . . . . . . . 13
4.2 Centralized DRL scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.3 Hotbooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..25
5.1 Performance versus Simulation Time . . . . . . . . . . . . . . . . . . . . . . 27
5.2 Varying number of CUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.3 Varying number of D2D TXs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.4 Performance with the Dynamic Environments . . . . . . . . . . . . . . . . 30
6 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32

[1] F. Jameel, Z. Hamid, F. Jabeen, S. Zeadally, and M. A. Javed, “A survey of device-to-device communications: Research issues and challenges,” IEEE Communications Surveys Tutorials, vol. 20, no. 3, pp. 2133–2168, 2018.
[2] K. Doppler, M. Rinne, C. Wijting, C. B. Ribeiro, and K. Hugl, “Device-to-device communication as an underlay to lte-advanced networks,” IEEE Communications Magazine, vol. 47, no. 12, pp. 42–49, 2009.
[3] T. Peng, Q. Lu, H. Wang, S. Xu, and W. Wang, “Interference avoidance mechanisms in the hybrid cellular and device-to-device systems,” in 2009 IEEE 20th International Symposium on Personal, Indoor and Mobile Radio Communications, 2009, pp. 617–621.
[4] P. Janis, V. Koivunen, C. Ribeiro, J. Korhonen, K. Doppler, and K. Hugl, “Interference-aware resource allocation for device-to-device radio underlaying cellular networks,” in VTC Spring 2009 - IEEE 69th Vehicular Technology Conference, 2009, pp. 1–5.
[5] X. Wu, Y. Chen, X. Yuan, and M. E. Mkiramweni, “Joint resource allocation and power control for cellular and device-to-device multicast based on cognitive radio,” IET Communications, vol. 8, no. 16, pp. 2805– 2813, 2014.
[6] H. Meshgi, D. Zhao, and R. Zheng, “Joint channel and power allocation in underlay multicast device-to-device communications,” in 2015 IEEE International Conference on Communications (ICC). IEEE, 2015, pp. 2937–2942.
[7] ——, “Optimal resource allocation in multicast device-to-device communications underlaying lte networks,” IEEE Transactions on Vehicular Technology, vol. 66, no. 9, pp. 8357–8371, 2017.
[8] H. Zhang, L. Song, and Z. Han, “Radio resource allocation for device-to-device underlay communication using hypergraph theory,” IEEE Transactions on Wireless Communications, vol. 15, no. 7, pp. 4852–4861, 2016.
[9] S. M. Alamouti and A. R. Sharafat, “Resource allocation for energy-efficient device-to-device communication in 4g networks,” in 7’th International Symposium on Telecommunications (IST’2014). IEEE, 2014, pp. 1058–1063.
[10] S. Dinh-Van, Y. Shin, and O.-S. Shin, “Resource allocation and power control based on user grouping for underlay device-to-device communications in cellular networks,” Transactions on Emerging Telecommunications Technologies, vol. 28, no. 1, p. e2920, 2017.
[11] M. Zhao, Y. Wei, M. Song, and G. Da, “Power control for d2d communication using multi-agent reinforcement learning,” in 2018 IEEE/CIC International Conference on Communications in China (ICCC). IEEE, 2018, pp. 563–567.
[12] S. Nie, Z. Fan, M. Zhao, X. Gu, and L. Zhang, “Q-learning based power control algorithm for d2d communication,” in 2016 IEEE 27th annual international symposium on personal, indoor, and mobile radio communications (PIMRC). IEEE, 2016, pp. 1–6.
[13] J. Tan, L. Zhang, and Y.-C. Liang, “Deep reinforcement learning for channel selection and power control in d2d networks,” in 2019 IEEE Global Communications Conference (GLOBECOM). IEEE, 2019, pp. 1–6.
[14] H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for v2v communications,” IEEE Transactions on Vehicular Technology, vol. 68, no. 4, 2019.
[15] R. O. Afolabi, A. Dadlani, and K. Kim, “Multicast scheduling and resource allocation algorithms for ofdma-based systems: A survey,” IEEE Communications Surveys Tutorials, vol. 15, no. 1, pp. 240–254, 2013.
[16] R. S. Sutton, A. G. Barto et al., “Introduction to reinforcement learning. vol. 135,” 1998.
[17] C. Weng, D. Yu, S. Watanabe, and B.-H. F. Juang, “Recurrent deep neural networks for robust speech recognition,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 5532–5536.
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[19] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
[20] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
[21] Z. Li and C. Guo, “Multi-agent deep reinforcement learning based spectrum allocation for d2d underlay communications,” IEEE Transactions on Vehicular Technology, vol. 69, no. 2, pp. 1828–1840, 2019.
[22] Y. Wei, F. R. Yu, M. Song, and Z. Han, “User scheduling and resource allocation in hetnets with hybrid energy supply: An actorcritic reinforcement learning approach,” IEEE Transactions on Wireless Communications, vol. 17, no. 1, pp. 680–692, 2017.
[23] C. Xia, S. Xu, and K. S. Kwak, “Resource allocation for device-to-device communication in lte-a network: A stackelberg game approach,” in 2014 IEEE 80th Vehicular Technology Conference (VTC2014-Fall). IEEE, 2014, pp. 1–5.
[24] 3GPP, “LTE; evolved universal terrestrial radio access (e-utra); physical layer procedures,” 3GPP, Technical specification (TS) 36.213, Sep 2020.
[25] F. Rebecchi, L. Valerio, R. Bruno, V. Conan, M. D. de Amorim, and A. Passarella, “A joint multicast/d2d learning-based approach to lte traffic offloading,” Computer Communications, vol. 72, pp. 26–37, 2015.
[26] X. Zhang, P. Yu, L. Feng, F. Zhou, and W. Li, “A drl-based resource allocation framework for multimedia multicast in 5g cellular networks,” in 2019 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB). IEEE, 2019, pp. 1–5.
[27] M. Hmila, M. Fern´andez-Veiga, M. Rodr´ıguez-P´erez, and S. Herrer´ıa- Alonso, “Energy efficient power and channel allocation in underlay device to multi device communications,” IEEE Transactions on Communications, vol. 67, no. 8, pp. 5817–5832, 2019.
[28] F. A. Onat, A. Adinoyi, Y. Fan, H. Yanikomeroglu, J. S. Thompson, and I. D. Marsland, “Threshold selection for snr-based selective digital relaying in cooperative wireless networks,” IEEE Transactions on Wireless Communications, vol. 7, no. 11, pp. 4226–4237, 2008.
[29] K. K. Nguyen, T. Q. Duong, N. A. Vien, N.-A. Le-Khac, and M.-N. Nguyen, “Non-cooperative energy efficient power allocation game in d2d communication: A multi-agent deep reinforcement learning approach,” IEEE Access, vol. 7, pp. 100 480–100 490, 2019.
[30] S.-Y. Chen, Y. Yu, Q. Da, J. Tan, H.-K. Huang, and H.-H. Tang, “Stabilizing reinforcement learning in dynamic environment with application to online recommendation,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1187–1196.
[31] L. Xiao, Y. Li, C. Dai, H. Dai, and H. V. Poor, “Reinforcement learning-based noma power allocation in the presence of smart jamming,” IEEE Transactions on Vehicular Technology, vol. 67, no. 4, pp. 3377–3389, 2017.
[32] M. Jung, K. Hwang, and S. Choi, “Joint mode selection and power allocation scheme for power-efficient device-to-device (d2d) communication,” in 2012 IEEE 75th vehicular technology conference (VTC Spring). IEEE, 2012, pp. 1–5.
[33] 3GPP, “Selection procedures for the choice of radio transmission technologies of the umts,” 3GPP, Technical report (TR) 30.03U, 1998.
[34] B. Kaufman and B. Aazhang, “Cellular networks with an overlaid device to device network,” in 2008 42nd Asilomar conference on signals, systems and computers. IEEE, 2008, pp. 1537–1541.