此論文探討單一無人機在無線感測網路中進行資料蒐集的飛行軌跡設計。相較以往的方法依序從感測器中一對一地蒐集資料，我們借助了多重存取通道中自然疊加的特性，來允許具有高度相關性的感測器在相同的通道中進行傳輸。我們分別針對離線和線上的情境提出以最小化資料重建均方誤差的無人機飛行策略。首先，在離線的設計中，我們透過考慮平均通道特性來最佳化整個蒐集時間內的無人機軌跡。對此，我們提出了離線版本的迭代演算法，藉由交替更新無人機軌跡和感測器傳輸的臨界值直到收斂為止。接下來，我們延伸到線上的場景，其中進一步考慮了飛行障礙物和短期衰退通道的效應。由於障礙物不僅會阻撓無人機的飛行，還會造成無人機和地面感測器之間存在非視距連接，我們設計了一種深度強化學習為基礎的線上飛行策略來針對複雜的環境，此策略可根據當前的估計誤差和即時通道條件來指導無人機下一步的移動。最後，我們透過數值模擬驗證了所提出之離線及線上無人機飛行策略對於感測器資料重建之有效性。

This work examines the UAV trajectory design for field estimation in wireless sensor networks using over-the-air sensor aggregation. Instead of collecting data from the sensors one-by-one, we allow sensors with highly correlated observations to transmit simultaneously over the same channel and utilize the superposition property of the multiple access channel to naturally aggregate their transmissions. Both offline and online UAV trajectory designs are proposed by minimizing the mean-squared error (MSE) of the reconstructed sensors' observations at the UAV. We first focus on an offline design that optimizes the UAV trajectory over the entire time horizon by considering average channel and sensor statistics. An iterative procedure is proposed for offline optimization where the UAV trajectory and the sensor selection threshold are updated in turn until convergence. Then, we extend to an online scenario, where flight obstacles and short-term channel effects are further taken into consideration. The flight obstacles impact not only the flight trajectory but also the presence or absence of line-of-sight links between UAV and ground sensors. A deep reinforcement learning algorithm is proposed to dynamically determine the movement depending on the current estimation error and immediate channel conditions. Numerical results are provided to demonstrate the effectiveness of our proposed schemes.

Abstract-----------i
Contents-----------ii
1. Introduction-----------1
2. System Model-----------7
3. Offline UAV Trajectory Optimization for MSE-Minimization-----------11
4. Online UAV Trajectory Design using Deep Reinforcement Learning-----------17
4.1 Markov Decision Process Formulation-----------19
4.2 Online UAV Navigation via a Deep Actor-Critic Network-----------23
5. Simulation Results-----------30
5.1 Offline Trajectory Design-----------30
5.2 Online Trajectory Design-----------34
6. Conclusion-----------40

[1] I. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci, “Wireless sensor networks: A survey,” Computer Networks, vol. 38, no. 4, pp. 393–422, 2002. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1389128601003024
[2] C. Konstantopoulos, G. Pantziou, D. Gavalas, A. Mpitziopoulos, and B. Mamalis, “A rendezvous-based approach enabling energy-efficient sensory data collection with mobile sinks,” IEEE Trans. Parallel Distrib. Syst., vol. 23, no. 5, pp. 809–817, 2012.
[3] Y. Yao, Q. Cao, and A. V. Vasilakos, “EDAL: An energy-efficient, delay-aware, and lifetime- balancing data collection protocol for heterogeneous wireless sensor networks,” IEEE/ACM Trans. Netw., vol. 23, no. 3, pp. 810–823, 2015.
[4] K. L.-M. Ang, J. K. P. Seng, and A. M. Zungeru, “Optimizing energy consumption for big data collection in large-scale wireless sensor networks with mobile collectors,” IEEE Syst. J., vol. 12, no. 1, pp. 616–626, 2018.
[5] J. Gong, T. Chang, C. Shen, and X. Chen, “Flight time minimization of UAV for data col- lection over wireless sensor networks,” IEEE J. Sel. Areas Commun., vol. 36, no. 9, pp. 1942–1954, Sep. 2018.
[6] C. Zhan, Y. Zeng, and R. Zhang, “Energy-efficient data collection in UAV-enabled wireless sensor network,” IEEE Wireless Commun. Lett., vol. 7, no. 3, pp. 328–331, June 2018.
[7] C. Zhan, Y. Zeng, and R. Zhang, “Trajectory design for distributed estimation in UAV- enabled wireless sensor network,” IEEE Trans. Veh. Technol., vol. 67, no. 10, pp. 10 155– 10 159, Oct 2018.
[8] K. Li, W. Ni, E. Tovar, and A. Jamalipour, “On-board deep Q-network for UAV-assisted online power transfer and data collection,” IEEE Trans. Veh. Technol., vol. 68, no. 12, pp. 12 215–12 226, 2019.
[9] M. Samir, S. Sharafeddine, C. M. Assi, T. M. Nguyen, and A. Ghrayeb, “UAV trajectory plan- ning for data collection from time-constrained IoT devices,” IEEE Trans. Wireless Commun., vol. 19, no. 1, pp. 34–46, 2020.
[10] C. You and R. Zhang, “3D trajectory optimization in Rician fading for UAV-enabled data harvesting,” IEEE Trans. Wireless Commun., vol. 18, no. 6, pp. 3192–3207, 2019.
[11] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial vehicles (UAVs) for energy-efficient internet of things communications,” IEEE Trans. Wireless Com- mun., vol. 16, no. 11, pp. 7574–7589, Nov 2017.
[12] O. M. Bushnaq, A. Celik, H. Elsawy, M.-S. Alouini, and T. Y. Al-Naffouri, “Aeronautical data aggregation and field estimation in iot networks: Hovering and traveling time dilemma of uavs,” IEEE Trans. Wireless Commun., vol. 18, no. 10, pp. 4620–4635, 2019.
[13] W. Luo, Y. Shen, B. Yang, S. Wang, and X. Guan, “Joint 3-d trajectory and resource opti- mization in multi-UAV-enabled IoT networks with wireless power transfer,” IEEE Internet Things J., vol. 8, no. 10, pp. 7833–7848, 2021.
[14] O. Bouhamed, H. Ghazzai, H. Besbes, and Y. Massoud, “A UAV-assisted data collection for wireless sensor networks: Autonomous navigation and scheduling,” IEEE Access, vol. 8, pp. 110 446–110 460, 2020.
[15] H. Bayerlein, M. Theile, M. Caccamo, and D. Gesbert, “UAV path planning for wireless data harvesting: A deep reinforcement learning approach,” in Proceedings of IEEE Global Commun. Conference (GLOBECOM), 2020, pp. 1–6.
[16] C. H. Liu, Z. Chen, and Y. Zhan, “Energy-efficient distributed mobile crowd sensing: A deep learning approach,” IEEE J. Sel. Areas Commun., vol. 37, no. 6, pp. 1262–1276, 2019.
[17] H. Bayerlein, M. Theile, M. Caccamo, and D. Gesbert, “Multi-UAV path planning for wire- less data harvesting with deep reinforcement learning,” IEEE Open J. Commun. Society, vol. 2, pp. 1171–1187, 2021.
[18] S.-H. Yeh, Y.-S. Wang, T. D. P. Perera, Y.-W. Peter Hong, and D. N. K. Jayakody, “UAV tra- jectory optimization for data-gathering from backscattering sensor networks,” in Proceedings of IEEE International Conference on Commun. (ICC), 2020, pp. 1–6.
[19] M. Gastpar, “Uncoded transmission is exactly optimal for a simple gaussian “sensor” net- work,” IEEE Trans. Inf. Theory, vol. 54, no. 11, pp. 5247–5251, 2008.
[20] S. Dasarathan and C. Tepedelenlioglu, “Distributed estimation and detection with bounded transmissions over gaussian multiple access channels,” IEEE Trans. Signal Process., vol. 62, no. 13, pp. 3454–3463, 2014.
[21] J. A. Maya, L. Rey Vega, and C. G. Galarza, “Optimal resource allocation for detection of a gaussian process using a MAC in WSNs,” IEEE Trans. Signal Process., vol. 63, no. 8, pp. 2057–2069, 2015.
[22] G. Zhu and K. Huang, “MIMO over-the-air computation for high-mobility multimodal sens- ing,” IEEE Internet Things J., vol. 6, no. 4, pp. 6089–6103, 2019.
[23] P. Zhang, I. Nevat, G. W. Peters, F. Septier, and M. A. Osborne, “Spatial field reconstruction and sensor selection in heterogeneous sensor networks with stochastic energy harvesting,” IEEE Trans. Signal Process., vol. 66, no. 9, pp. 2245–2257, May 2018.
[24] A. Sani and A. Vosoughi, “Distributed vector estimation for power- and bandwidth- constrained wireless sensor networks,” IEEE Trans. Signal Process., vol. 64, no. 15, pp. 3879–3894, Aug 2016.
[25] A. Al-Hourani, S. Kandeepan, and A. Jamalipour, “Modeling air-to-ground path loss for low altitude platforms in urban environments,” in Proceedings of IEEE Global Commun. Conference (GLOBECOM), 2014, pp. 2898–2904.
[26] A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum cover- age,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, Dec 2014.
[27] R. I. Bor-Yaliniz, A. El-Keyi, and H. Yanikomeroglu, “Efficient 3D placement of an aerial base station in next generation cellular networks,” in Proceedings of IEEE International Con- ference on Commun. (ICC), 2016, pp. 1–5.
[28] K. Shen and W. Yu, “Fractional programming for communication systems - Part II: uplink scheduling via matching,” IEEE Trans. Signal Process., vol. 66, no. 10, pp. 2631–2644, May 2018.
[29] A. Beck, “On the convergence of alternating minimization for convex programming with applications to iteratively reweighted least squares and decomposition schemes,” SIAM J. Optim., vol. 25, pp. 185–209, 2015.
[30] S. Bahmani and B. Raj, “A unifying analysis of projected gradient descent for {!p- constrained least squares,” Applied and Computational Harmonic Analysis, vol. 34, no. 3,
pp. 366–378, 2013. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S106352031200108X
[31] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. USA: Prentice-Hall, Inc., 1993.
[32] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. The MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html
[33] C. J. C. H. Watkins and P. Dayan, “Q-learning,” in Machine Learning, 1992, pp. 279–292.

[34] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Ried- miller, “Playing atari with deep reinforcement learning,” Available: arXiv: 1312.5602, 2013.
[35] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” Available: arXiv: 1509.02971, 2015.
[36] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic pol- icy gradient algorithms,” in Proceedings of International Conference on Machine Learning (ICML), vol. 1, June 2014.
[37] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan,
A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys, “Deep Q- learning from demonstrations,” in Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2018.
[38] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rotho¨rl,
T. Lampe, and M. Riedmiller, “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,” Available: arXiv:1707.08817, 2017.
[39] R. Ding, F. Gao, and X. S. Shen, “3d uav trajectory design and frequency band allocation for energy-efficient and fair communication: A deep reinforcement learning approach,” IEEE Trans. Wireless Commun., vol. 19, no. 12, pp. 7796–7809, 2020.