本論文探討了天-空一體化網路中混合雲及邊緣雲之計算服務。在此網路中，無人機（UAV）提供邊緣計算服務，而衛星（SAT）則提供雲計算服務。為了高效地提供這些服務，需要考慮到UAV和SAT之可用計算資源、能量和通訊資源等限制。因此，本論文基於此場景設計了一個任務調度及資源分配問題，旨在最小化所有任務之計算與卸載總時延。
為了解決此問題，本論文提出了一種演算法，將原問題分解為兩個可交替執行的子問題：計算資源分配以及任務調度。其中，計算資源分配的子問題可基於凸優化方法求解（涉及實變數），用於最佳化分配UAV和SAT的計算資源；而任務調度的子問題則是基於雙深度Q學習（DDQL）獲得有效調度方案（涉及二元變數），用於高效協調安排任務給所有的UAV和SAT。
研究結果表明，本論文提出的方法相較於僅採用深度強化學習演算法的相關研究，不僅在訓練階段有更短的算法運行時間和更佳的系統可擴展性，同時，在測試階段的計與和卸載總時延方面表現也更為優越。這些優勢在文中之多組實驗模擬中均得到了驗證。因此，在天-空一體化網絡中，該算法可以有效分配雲及邊緣雲計算資源，並且兼顧實際應用層面的計算效率，從而提高系統服務的性能，對於未來的相關研究及應用提供了良好的參考。

This thesis investigates space-aerial assisted mixed cloud-edge computing services for space-aerial integrated networks (SAINs), where unmanned aerial vehicles (UAVs) provide edge computing services and one satellite (SAT) provides ubiquitous cloud computing services. To effectively and efficiently schedule such services under constraints on the available resources of computational capacity, energy, and communications of UAVs and the SAT, a problem for minimizing the total computing and offloading delay is formulated. A learning algorithm for handling the reformulated problem is proposed that alternatively performs convex optimization based computational capacity allocation (involving real variables) and double deep Q-learning (DDQL) based task assignment (involving binary variables) among all UAVs and the SAT. Extensive simulation results are presented to demonstrate that the efficacy of the proposed algorithm is significantly superior over some state-of-the-art reinforcement learning (RL) based methods in terms of the algorithm running time and system scalability in the training stage and total computing and offloading delay in the testing stage.
computing and offloading delay is formulated. A learning algorithm for handling
the reformulated problem is proposed that alternatively performs convex opti-
mization based computational capacity allocation (involving real variables) and
double deep Q-learning (DDQL) based task assignment (involving binary vari-
ables) among all UAVs and the SAT. Extensive simulation results are presented
to demonstrate that the efficacy of the proposed algorithm is significantly superior
over some state-of-the-art reinforcement learning (RL) based methods in terms of
the algorithm running time and system scalability in the training stage and total
computing and offloading delay in the testing stage.

Table of Contents
Abstract (Chinese) . . . i
Abstract . . . ii
Acknowledgements (Chinese) . . . iii
Table of Contents . . . iv
List of Figures . . . vi
List of Tables . . . viii
1 Introduction . . . 1
2 System Model . . . 5
2.1 Network Model . . . 5
2.2 Task Model . . . 7
2.2.1 UAV Edge Computing . . . 7
2.2.2 Task Offloading . . . 7
2.2.3 SAT Cloud Computing . . . 8
2.2.4 Problem Formulation . . . 9
3 Proposed Optimization Method 11
3.1 Size-reduced Problem Reformulation . . . 11
3.2 Markov Decision Process (MDP) Formulation . . . 13
3.2.1 State . . . 14
3.2.2 Action . . . 14
3.2.3 Reward Function . . . 14
3.2.4 Comparison of RL methods: QL, DQL, and DDQL . . . 15
3.2.5 DDQL Algorithm for Solving P3 (DDQL-P3) in Training Stage . . . 16
3.2.6 DDQL Algorithm for Solving P3 (DDQL-P3) in Testing Stage . . . 17
4 Simulation Results and Discussions . . . 20
4.1 Simulation Settings . . . 20
4.2 Simulation Results . . . 22
5 Conclusions . . . 27
Bibliography . . . 32

[1] N. Fernando, S. W. Loke, and W. Rahayu, “Mobile cloud computing: A
survey,” Future Generation Computer Systems, vol. 29, no. 1, pp. 84–106, 2013.
[2] C. Wang, C. Liang, F. R. Yu, Q. Chen, and L. Tang, “Computation offloading and resource allocation in wireless cellular networks with mobile edge computing,” IEEE Trans. Wireless Communications, vol. 16, no. 8, pp. 4924–4938, 2017.
[3] F. Fang, Y. Xu, Z. Ding, C. Shen, M. Peng, and G. K. Karagiannidis, “Optimal resource allocation for delay minimization in NOMA-MEC networks,” IEEE Trans. Communications, vol. 68, no. 12, pp. 7867–7881, 2020.
[4] N. Waqar, S. A. Hassan, A. Mahmood, K. Dev, D.-T. Do, and M. Gidlund, “Computation offloading and resource allocation in MEC-enabled integrated aerial-terrestrial vehicular networks: A reinforcement learning approach,” IEEE Trans. Intelligent Transportation Systems, vol. 1, pp. 1–14, 2022.
[5] Y. Li, S. Wang, C.-Y. Chi, and T. Q. Quek, “Differentially private federated learning in edge networks: The perspective of noise reduction,” IEEE Network, vol. 36, no. 5, pp. 167–172, 2022.
[6] S. Mao, S. He, and J. Wu, “Joint UAV position optimization and resource scheduling in space-air-ground integrated networks with mixed cloud-edge computing,” IEEE Systems Journal, vol. 15, no. 3, pp. 3992–4002, 2020.
[7] N. Cheng, F. Lyu, W. Quan, C. Zhou, H. He, W. Shi, and X. Shen,
“Space/aerial-assisted computing offloading for IoT applications: A learning-based approach,” IEEE Journal of Selected Areas in Communications, vol. 37, no. 5, pp. 1117–1129, 2019.
[8] C. Liu, W. Feng, Y. Chen, C.-X. Wang, and N. Ge, “Cell-free satellite-UAV networks for 6G wide-area Internet of Things,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 4, pp. 1116–1131, 2020.
[9] C. Lei, W. Feng, Y. Chen, and N. Ge, “Joint power and channel allocation for safeguarding cognitive satellite-UAV networks,” in Proc. IEEE Global Communications Conference (GLOBECOM), 2021, pp. 1–6.
[10] K.-C. Tsai, L. Fan, L.-C. Wang, R. Lent, and Z. Han, “Multi-commodity flow routing for large-scale LEO satellite networks using deep reinforcement learning,” in Proc. IEEE Wireless Communications and Networking Conference (WCNC), 2022, pp. 626–631.
[11] K.-C. Tsai, T.-J. Yao, P.-H. Huang, C.-S. Huang, Z. Han, and L.-C. Wang, “Deep reinforcement learning-based routing for space-terrestrial networks,” in Proc. IEEE 96th Vehicular Technology Conference (VTC2022-Fall), 2022, pp. 1–5.
[12] Y. Zhou, N. Cheng, N. Lu, and X. S. Shen, “Multi-UAV-aided networks: Aerial-ground cooperative vehicular networking architecture,” IEEE Vehicular Technology Magazine, vol. 10, no. 4, pp. 36–44, 2015.
[13] N. Cheng, W. Xu, W. Shi, Y. Zhou, N. Lu, H. Zhou, and X. Shen, “Air-ground
integrated mobile edge networks: Architecture, challenges, and opportunities,” IEEE Communications Magazine, vol. 56, no. 8, pp. 26–32, 2018.
[14] Y. Hu and V. O. Li, “Satellite-based Internet: A tutorial,” IEEE Communications Magazine, vol. 39, no. 3, pp. 154–162, 2001.
[15] S. Wang, S. Bi, and Y.-J. A. Zhang, “Deep reinforcement learning with communication transformer for adaptive live streaming in wireless edge networks,” IEEE Journal of Selected Areas in Communications, vol. 40, no. 1, pp. 308-322, 2022.
[16] J. Zhao, Q. Li, Y. Gong, and K. Zhang, “Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks,” IEEE Trans. Vehicular Technology, vol. 68, no. 8, pp. 7944–7956, 2019.
[17] J. Du, L. Zhao, J. Feng, and X. Chu, “Computation offloading and resource allocation in mixed fog/cloud computing systems with min-max fairness guarantee,” IEEE Trans. Communications, vol. 66, no. 4, pp. 1594–1608, 2017.
[18] C. Zhou, W. Wu, H. He, P. Yang, F. Lyu, N. Cheng, and X. Shen, “Delay-aware IoT task scheduling in space-air-ground integrated network,” in Proc. IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6.
[19] A. D. Panagopoulos, P.-D. M. Arapoglou, and P. G. Cottis, “Satellite communications at Ku, Ka, and V bands: Propagation impairments and mitigation techniques,” IEEE Communications Surveys & Tutorials, vol. 6, no. 3, pp.2–14, 2004.
[20] C.-Y. Chi, W.-C. Li, and C.-H. Lin, Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications. Boca Raton, FL, USA: CRC Press, 2017.
[21] M. A. Wiering and M. Van Otterlo, “Reinforcement learning,” Adaptation, Learning, and Optimization, vol. 12, no. 3, p. 729, 2012.
[22] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
[23] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proc. AAAI Conference on Artificial Intelligence, 2016, pp. 1–7.
[24] H. Zhou, K. Jiang, X. Liu, X. Li, and V. C. M. Leung, “Deep reinforcement learning for energy-efficient computation offloading in mobile-edge computing,” IEEE Internet of Things Journal, vol. 9, no. 2, pp. 1517–1530, 2022.