這項研究提出了一種新穎的網路內學習框架，用於物聯網(IoT)中的分散式推理，其中低成本感測器裝置可以通過互相合作在無線多躍式組成深度神經網路。這些設備分為多個層，包括一個源層、多個中繼層和一個目標層。每個單獨的裝置僅由一個小型神經網路子模型組成，但將它們聚集起來能夠執行複雜的推理任務，當資料以逐層方式逐漸傳遞到目的地。藉由利用多址通道的疊加特性，我們提出使用空中計算在有限數量的時槽內有效地執行神經網路層之間所需的線性組合操作。基於感測器的層分配，提出了兩種方法來設計發射和接收濾波器以實現空中計算。在第一種方法中，濾波器藉由基於最小均方誤差(MMSE)標準來逼近集中式神經網路的權重參數。在第二種方法中，濾波器是藉由直接對不同通道進行端到端訓練獲得的。然後，提出了一種基於網路剪枝的層分配策略來確定與不同層關聯的傳感器。實驗結果顯示，隨著訊噪比和時槽數量的增加，我們提出的網路內學習框架能夠實現與集中式模型接近的推理準確度。

This work proposes a novel in-network learning framework for distributed inference in internet-of-things (IoT), where low-cost sensor devices may collaborate to form a deep neural network over wireless multihop links.
The devices are grouped into multiple layers, including a source layer, several relay layers, and a destination layer.
Each individual device consists of only a small neural network submodel, but their aggregate is capable of performing complex inference tasks as local observations are gradually forwarded in a layer-by-layer fashion to the destination.
By leveraging the superposition property of the multiple access channel, we propose the use of over-the-air computation to efficiently perform the linear combining operations required between neural network layers over a limited number of time slots.
Given the layer assignments of the sensors, two approaches are proposed for the design of the transmit and receive filters to enable over-the-air computation. In the first approach, the filters are designed to approximate the weight parameters of a centralized neural network based on the minimum mean squared error (MMSE) criterion. In the second approach, the filters are directly obtained by end-to-end training over varying channel realizations. Then, a layer assignment policy is proposed based on network pruning to determine the sensors associated with different layers. Simulation results show that our proposed in-network learning framework is able to achieve an inference accuracy close to that of the centralized model as the signal-to-noise ratio and the number of time slots increase.

Abstract i
Contents iii
1 Introduction 1 2 Related Works 5
3 System Model 10
4 Transmit and Recieve Filter Designs for Over-the-Air In-Network Learn-
ing 15
4.1 Approach I: MMSE-based Design . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Approach II: End-to-End Training Design . . . . . . . . . . . . . . . . . . . 18
5 Layer Assignment by Network Pruning 22
6 Experimental Result 26
6.1 Experiments for In-Network Learning under Fixed Layer Assignment . . . . 27
6.2 Experiments for Layer Assignment by Network Pruning . . . . . . . . . . . . 35
7 Conclusion 46
Bibliography 47

[1] S. Yao, Y. Zhao, A. Zhang, S. Hu, H. Shao, C. Zhang, L. Su, and T. Abdelzaher, “Deep learning for the internet of things,” Computer, vol. 51, no. 5, pp. 32–41, 2018.
[2] H. Li, K. Ota, and M. Dong, “Learning IoT in edge: Deep learning for the internet of things with edge computing,” IEEE Network, vol. 32, no. 1, pp. 96–101, 2018.
[3] J.-J. Xiao, A. Ribeiro, Z.-Q. Luo, and G. B. Giannakis, “Distributed compressionestimation using wireless sensor networks,” IEEE Signal Process. Mag., vol. 23, no. 4, pp. 27–41, 2006.
[4] J. Chamberland and V. V. Veeravalli, “Wireless sensors in distributed detection applications,” IEEE Signal Process. Mag., vol. 24, no. 3, pp. 16–25, 2007.
[5] X. Wang, Y. Han, V. C. M. Leung, D. Niyato, X. Yan, and X. Chen, “Convergence of edge computing and deep learning: A comprehensive survey,” IEEE Commun. Surveys Tuts., vol. 22, no. 2, pp. 869–904, 2020.
[6] S. Yao, Y. Zhao, A. Zhang, L. Su, and T. Abdelzaher, “DeepIoT: Compressing deep neural network structures for sensing systems with a compressor-critic framework,” in Proceedings of ACM Conference on Embedded Network Sensor Systems (SenSys), 2017.
[7] S. Fu, Z. Li, K. Liu, S. Din, M. Imran, and X. Yang, “Model compression for IoT applications in industry 4.0 via multiscale knowledge transfer,” IEEE Trans. Ind. Informat., vol. 16, no. 9, pp. 6013–6022, 2020.
[8] R. Du, S. Magnusson, and C. Fischione, “The internet of things as a deep neural network,” IEEE Commun. Mag., vol. 58, no. 9, pp. 20–25, 2020.
[9] Z. Zhao, K. M. Barijough, and A. Gerstlauer, “DeepThings: Distributed adaptive deep learning inference on resource-constrained IoT edge clusters,” IEEE Trans. Comput.Aided Design Integr. Circuits Syst., vol. 37, no. 11, pp. 2348–2359, 2018.
[10] B. Nazer and M. Gastpar, “Computation over multiple-access channels,” IEEE Trans. Inf. Theory, vol. 53, no. 10, pp. 3498–3516, 2007.
[11] M. Goldenbaum, H. Boche, and S. Stan´czak, “Harnessing interference for analog function computation in wireless sensor networks,” IEEE Trans. Signal Process., vol. 61, no. 20, pp. 4893–4906, 2013.
[12] Y.-W. Peter Hong and C.-C. Wang, “In-network learning via over-the-air computation in internet-of-things,” in Proceedings of IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), 2021, pp. 141–145.
[13] O. Gupta and R. Raskar, “Distributed learning of deep neural network over multiple agents,” J. Network and Computer Applications, vol. 116, pp. 1–8, 2018.
[14] C. Lee, J. Lin, P. Chen, and Y. Chang, “Deep learning-constructed joint transmissionrecognition for internet of things,” IEEE Access, vol. 7, pp. 76547–76561, 2019.
[15] E. Bourtsoulatze, D. Burth Kurka, and D. Gu¨ndu¨z, “Deep joint source-channel coding for wireless image transmission,” IEEE Trans. on Cogn. Commun. Netw., vol. 5, no. 3, pp. 567–579, 2019.
[16] K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation,” IEEE Trans. Wireless Commun., vol. 19, no. 3, pp. 2022–2035, 2020.
[17] S. Teerapittayanon, B. McDanel, and H. Kung, “Distributed deep neural networks over the cloud, the edge and end devices,” in Proceedings of IEEE International Conference on Distributed Computing Systems (ICDCS), 2017, pp. 328–339.
[18] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” in Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Apr. 2017, p. 615–629.
[19] S. Zhang, S. Zhang, Z. Qian, J. Wu, Y. Jin, and S. Lu, “DeepSlicing: Collaborative and adaptive cnn inference with low latency,” IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 9, pp. 2175–2187, 2021.
[20] Y. Hu, D. Niu, J. Yang, and S. Zhou, “FDML: A collaborative machine learning framework for distributed features,” in Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2019, p. 2232–2240.
[21] J. Kim, Y. Park, G. Kim, and S. J. Hwang, “SplitNet: Learning to semantically split deep networks for parameter reduction and model parallelization,” in Proceedings of International Conference on Machine Learning (ICML), vol. 70, 2017, pp. 1866–1874.
[22] J. Huang, C. Samplawski, D. Ganesan, B. Marlin, and H. Kwon, “CLIO: Enabling automatic compilation of deep learning pipelines across IoT and cloud,” in Proceedings of Annual International Conference on Mobile Computing and Networking (MobiCom),
2020.
[23] S. Tuli, G. Casale, and N. R. Jennings, “SplitPlace: AI augmented splitting and placement of large-scale neural networks in mobile edge environments,” IEEE Trans. Mob. Comput. (Early Access), 2022.
[24] M. Gastpar, “Uncoded transmission is exactly optimal for a simple Gaussian sensor network,” IEEE Trans. Inf. Theory, vol. 54, no. 11, pp. 5247–5251, 2008.
[25] S. Dasarathan and C. Tepedelenlioglu, “Distributed estimation and detection with bounded transmissions over Gaussian multiple access channels,” IEEE Trans. Signal Process., vol. 62, no. 13, pp. 3454–3463, 2014.
[26] G. Zhu and K. Huang, “MIMO over-the-air computation for high-mobility multimodal sensing,” IEEE Internet Things J., vol. 6, no. 4, pp. 6089–6103, 2019.
[27] D. Wen, G. Zhu, and K. Huang, “Reduced-dimension design of MIMO over-the-air computing for data aggregation in clustered IoT networks,” IEEE Trans. Wireless Commun., vol. 18, no. 11, pp. 5255–5268, 2019.
[28] G. Zhu, J. Xu, K. Huang, and S. Cui, “Over-the-air computing for wireless data aggregation in massive IoT,” IEEE Wireless Commun., vol. 28, no. 4, pp. 57–65, 2021.
[29] G. Zhu, Y. Wang, and K. Huang, “Broadband analog aggregation for low-latency federated edge learning,” IEEE Trans. Wireless Commun., vol. 19, no. 1, pp. 491–506, 2020.
[30] M. M. Amiri and D. Gu¨ndu¨z, “Federated learning over wireless fading channels,” IEEE
Trans. Wireless Commun., vol. 19, no. 5, pp. 3546–3557, 2020.
[31] T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical layer,”
IEEE Trans. on Cogn. Commun. Netw., vol. 3, no. 4, pp. 563–575, 2017.
[32] A. Felix, S. Cammerer, S. Do¨rner, J. Hoydis, and S. Ten Brink, “OFDM-autoencoder for end-to-end learning of communications systems,” in Proceedings of IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), 2018.
[33] T. J. O’Shea, T. Erpek, and T. C. Clancy, “Physical layer deep learning of encodings for the mimo fading channel,” in Proceedings of the Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2017, pp. 76–80.
[34] H. Ye, G. Y. Li, and B.-H. Juang, “Deep learning based end-to-end wireless communication systems without pilots,” IEEE Trans. on Cogn. Commun. Netw., vol. 7, no. 3, pp. 702–714, 2021.
[35] H. Ye, L. Liang, G. Y. Li, and B.-H. Juang, “Deep learning-based end-to-end wireless communication systems with conditional GANs as unknown channels,” IEEE Trans. Wireless Commun., vol. 19, no. 5, pp. 3133–3143, 2020.
[36] F. A. Aoudia and J. Hoydis, “Model-free training of end-to-end communication systems,” IEEE J. Sel. Areas Commun., vol. 37, no. 11, pp. 2503–2516, 2019.
[37] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” IEEE Trans. Signal Process., vol. 69, pp. 2663–2675, 2021.
[38] L. Liu, A. Solomon, S. Salamatian, and M. M´edard, “Neural network coding,” in Proceedings of IEEE International Conference on Communications (ICC), 2020.
[39] X. Zhai, X. Chen, J. Xu, and D. W. Kwan Ng, “Hybrid beamforming for massive MIMO over-the-air computation,” IEEE Transactions on Communications, vol. 69, no. 4, pp.
2737–2751, 2021.
[40] M. Soltanalian, M. M. Naghsh, N. Shariati, P. Stoica, and B. Hassibi, “Training signal design for correlated massive MIMO channel estimation,” IEEE Transactions on Wireless Communications, vol. 16, no. 2, pp. 1135–1143, 2017.
[41] M. Goldenbaum and S. Stanczak, “On the channel estimation effort for analog computation over wireless multiple-access channels,” IEEE Wireless Communications Letters, vol. 3, no. 3, pp. 261–264, 2014.
[42] K. Lee, H. Kim, H. Lee, and D. Shin, “Flexible group-level pruning of deep neural networks for on-device machine learning,” in Proceedings of Design, Automation Test in Europe Conference Exhibition (DATE), 2020, pp. 79–84.
[43] Y. LeCun, “The MNIST database of handwritten digits,” http://yann. lecun. com/exdb/mnist/, 1998.
[44] X. Cao, G. Zhu, J. Xu, and K. Huang, “Optimized power control for over-the-air computation in fading channels,” IEEE Trans. Wireless Commun., vol. 19, no. 11, pp. 7498–7513, 2020.
[45] L. Lu, “Dying ReLU and initialization: Theory and numerical examples,” Communications in Computational Physics, vol. 28, no. 5, pp. 1671–1706, jun 2020.
[Online]. Available: https://doi.org/10.4208%2Fcicp.oa-2020-0165