低軌道衛星網路通常由數百顆以上的低軌道衛星組成，並且提供了廣域的覆蓋，特
別是在偏遠地區扮演著重要的角色。一旦用戶終端離開其服務衛星的覆蓋範圍，該
衛星便無法繼續為這個用戶提供服務，因此用戶必須換手至其他衛星。然而，低軌
道衛星的高速運行會引起頻繁的換手，從而影響用戶的連線品質。此外，用戶數量
的增加以及一顆衛星中有限的頻道數量，使得資源分配成為一個具有挑戰性的問
題。由於上述問題，如何選擇適當的換手目標變得非常重要。
在本文中，我們採用雙連線的環境並且提出了一種結合序列到序列模型和強化
學習的方法來解決這個問題。我們利用序列到序列模型的特性，並將換手問題轉
化為多標籤分類問題，從而實現了最大化用戶Quality of Experience（QoE）的目
標。實驗結果表明，與貪婪演算法相比，我們的方法在用戶的平均QoE方面具有更
好的表現。

Low Earth orbit (LEO) satellite networks often consist of more than a few hundred LEO satellites and play a key role in providing wide-area coverage, particularly for remote areas. Once a user terminal (UT) leaves the coverage area of its serving satellite, the satellite cannot provide services to the UT, therefore a handover to other satellite needs to be performed. However, the fast orbital speed of LEO satellites would result in frequent handovers, which affect the quality of the UT’s connection. Additionally, the increasing number of UTs and the limited number of channels of a satellite make resource allocation a challenging problem. Due to the above problems, how to select appropriate handover targets is very important.
In the thesis, we adopt a dual connectivity environment and propose a method combining a Sequence-to-Sequence (Seq2Seq) model with reinforcement learning (RL) to solve this problem. We achieve our goal of maximizing the Quality of Experience (QoE) for UTs by leveraging the characteristics of the Seq2Seq model and transforming the handover problem into a multi-label classification problem. The simulation results show that our method outperforms the greedy method in terms of the average QoE for UTs.

Acknowledgements i
摘要 ii
Abstract iii
Contents iv
List of Figures vi
List of Tables viii
1 Introduction 1
2 Related Works 3
3 Preliminary 5
3.1 5G Satellite Network . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.1 Three Types of Satellite Orbits . . . . . . . . . . . . . . . . . 5
3.1.2 Mobile Satellite Networks (MSNs) . . . . . . . . . . . . . . . . 6
3.2 Packet Error Rate (PER) . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Reinforcement Learning (RL) . . . . . . . . . . . . . . . . . . . . . . 8
3.3.1 Background of RL . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3.2 Proximal Policy Optimization (PPO) . . . . . . . . . . . . . . 9
iv
3.4 Sequence-to-Sequence Model (Seq2Seq) . . . . . . . . . . . . . . . . . 10
3.4.1 Background of Seq2Seq . . . . . . . . . . . . . . . . . . . . . . 10
4 The Problem 12
4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5 The Proposed Method 13
5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.2 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6 Simulation 17
6.1 The Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
6.2 Performance metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
6.3 Comparison method . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
6.4 The Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
7 Conclusion 21
References 22

[1] Z.-H. Huang, C.-Y. Huang, and M.-J. Tsai, “Efficient multi-connectivity handover algorithm in heterogeneous cellular networks by graph-to-sequence reinforcement learning,” in GLOBECOM 2023 - 2023 IEEE Global Communications Conference, pp. 7423–7428, 2023.
[2] M. Sana, A. D. Domenico, E. C. Strinati, and A. Clemente, “Multi-agent deep reinforcement learning for distributed handover management in dense mmwave networks,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8976–8980, 2020.
[3] Y. Chen, X. Lin, T. Khan, and M. Mozaffari, “Efficient drone mobility support using reinforcement learning,” in 2020 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6, 2020.
[4] S. He, T. Wang, and S. Wang, “Load-aware satellite handover strategy based on multi-agent reinforcement learning,” in GLOBECOM 2020 - 2020 IEEE Global Communications Conference, pp. 1–6, 2020.
[5] J. Wang, W. Mu, Y. Liu, L. Guo, S. Zhang, and G. Gui, “Deep reinforcement learning-based satellite handover scheme for satellite communications,” in 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1–6, 2021.
[6] J. Yang, Z. Xiao, S. Member, H. Cui, J. Zhao, G. Jiang, and Z. Han, “DQN-alrm-based intelligent handover method for satellite-ground integrated network,” in IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS
AND NETWORKING, vol. 9, no. 4, pp. 977–990, 2023.
[7] H. Liu, Y. Wang, and Y. Wang, “A successive deep q-learning based distributed handover scheme for large-scale LEO satellite networks,” in 2022 IEEE 95th Vehicular Technology Conference (VTC2022-Spring), pp. 1–6, 2022.
[8] N. Badini, M. Jaber, M. Marchese, and F. Patrone, “Reinforcement learningbased load balancing satellite handover using ns-3,” in ICC 2023 - IEEE International Conference on Communications, pp. 2595–2600, 2023.
[9] H. Xu, D. Li, M. Liu, G. Han, S. Member, W. Huang, and C. Xu, “QoE-driven intelligent handover for user-centric mobile satellite networks,” in IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, vol. 69, no. 9, p. 10127–10139, 2020.
[10] “Study on New Radio (NR) to support non-terrestrial networks,” 3GPP TR 38.811 version 15.1.0 Release 15, 2019.
[11] “Ansys STK Software for Digital Mission Engineering and Systems Analysis.” https://www.ansys.com/products/missions/ansys-stk.
[12] “5G; System Architecture for the 5G System,” 3GPP TS 23.501 version 15.2.0 Release 15, 2020.