深度神經網絡(Deep Neural Network)被廣泛用於分析邊端設備感測器收集 的大量數據，目前最常見的方法是將資料發送到雲端上，並在運算力強大的雲端 伺服器上進行 DNN 推論。但是僅依靠雲端的方法會遭受較長的網路通信延遲， 而且有隱私問題。近年來，邊端運算的概念被提出，並被用作於將原本在雲端的 運算移動到邊端，藉此減少網路流量和雲端的負載，許多研究試圖將 DNN 推論 完全或部分的卸載到邊緣設備，但是單一個資源有限的邊端設備並沒有足夠能力 即時進行 DNN 推斷，因此，我們開發了一種合作式邊端運算架構來將 DNN 推 論分散到多個邊端設備上，並且提出一個 DNN 分層切割演算法，稱之為 MLS， 用以達到最小化執行時間。根據實驗結果顯示，與過去的方法相比，我們的演算 法有較快速的執行時間。

Deep Neural Networks (DNNs) are widely used to analyze the abundance of data collected by edge device sensors. The stat-of-the-art approach is to send the data to the cloud and process the DNN inference on the powerful cloud servers. However, cloud- only method suffers from long network communication latency and raises privacy concerns. In recent years, the concept of edge computing is proposed to reduce the network traffic and the cloud load by moving computation from the cloud to the edges. Many researches try to fully or partially localize the DNN inference to edge devices. However, a single resource-constrained edge device is not powerful enough to process a DNN inference in real-time. Thus, we develop a collaborative edge computing framework to distribute DNN inference to several edge devices. And we propose a multi-layer slicing (MLS) algorithm to minimize the execution time of the distributed DNN inference. The evaluation results show that our algorithm has better performance on execution time than previous work.

1. Introduction .......................................................................................................... 1
2. Related Work........................................................................................................4
3. Background .......................................................................................................... 8
3.1 Data/Intra-model parallelism ...........................................................................8
3.2 Layer Slicing..................................................................................................10
4. Multi-Layer Slicing Algorithms (MLS) ........................................................... 13
4.1 Problem Formulation .....................................................................................13
4.2 Presentation of Proposed MLS Algorithm.....................................................15
4.3 Building Block Execution Time Table...........................................................17
5. Performance Evaluation....................................................................................23
5.1 Changing Number of Devices on RP3...........................................................27
5.2 Changing Number of Devices on RP4...........................................................30
5.3 Changing Network Bandwidth ......................................................................32
5.4 Changing flops of the Edge Devices..............................................................33
6. Conclusion .......................................................................................................... 35
References ................................................................................................................... 36 Appendix A .................................................................................................................39

[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105, Lake Tahoe, Nevada, USA, Dec. 2012.
[2] G. Hinton et al., "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, Nov. 2012.
[3] R. Sarikaya, G. E. Hinton, and A. Deoras, "Application of Deep Belief Networks for Natural Language Understanding," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 4, pp. 778-784, April 2014.
[4] S. A. Osia, A. S. Shamsabadi, A. Taheri, H. R. Rabiee, and H. Haddadi, "Private and Scalable Personal Data Analytics Using Hybrid Edge-to-Cloud Deep Learning," IEEE Computer, vol. 51, no. 5, pp. 42-49, May 2018.
[5] R. Shokri and V. Shmatikov, "Privacy-Preserving Deep Learning," Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1310–1321, Denver, Colorado, USA, Oct. 2015.
[6] S. A. Osia, A. S. Shamsabadi, A. Taheri, K. Katevas, H. R. Rabiee, N. D. Lane, and H. Haddadi," Privacy-preserving deep inference for rich user data on the cloud," arXiv preprint arXiv:1710.01727, Oct. 2017.
[7] J. Chi et al., "Privacy Partition: A Privacy-Preserving Framework for Deep Neural Networks in Edge Networks," Proceedings of the IEEE/ACM Symposium on Edge Computing, pp.378-380, Bellevue, WA, USA, Oct. 2018.
[8] B. Zhang et al., "The cloud is not enough: Saving IoT from the cloud," USENIX Workshop on Hot Topics in Cloud Computing, Santa Clara, CA, USA, Jul. 2015.
[9] The Growth in Connected IoT Devices Is Expected to Generate 79.4ZB of Data
in 2025, According to a New IDC Forecast. https://www.idc.com/getdoc.jsp?containerId=prUS45213219. Accessed:2019-
10
[10] P. Garcia Lopez et al., "Edge-centric Computing: Vision and Challenges," ACM
SIGCOMM Computer Communication Review, vol. 45, no. 5, pp. 37–42, Oct.
2015.
[11] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, "Edge Computing: Vision and
Challenges," IEEE Internet of Things Journal, Vol. 3, No. 5, pp. 637-646, Oct. 2016.
[12] K. Bierzynski, A. Escobar, and M. Eberl, "Cloud, Fog and Edge: Cooperation for the Future?" Proceedings of the second International Conference on Fog and Mobile Edge Computing, pp. 62-67, Valencia, Spain, May 2017
[13] S. Yao, Y. Zhao, H. Shao, S. Liu, D. Liu, L. Su, and T. Abdelzaher, "FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices," Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, pp. 278–291, Shenzhen, China, Nov. 2018.
[14] S. Han, H. Mao, and W. J Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman coding," arXiv preprint arXiv:1510.00149, Oct. 2015.
[15] K. Bhardwaj, C. Lin, A. Sartor, and R. Marculescu, "Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT," arXiv preprint arXiv:1907.11804, 2019.
[16] J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke, "Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism," Proceedings of the ACM/IEEE 44th Annual International Symposium on Computer Architecture, pp. 548–560, Toronto, ON, Canada, Jun. 2017.
[17] S. Teerapittayanon, B. McDanel, and H. T. Kung, "Distributed Deep Neural Networks Over the Cloud, the Edge and Edge devices," Proceedings of the IEEE 37th International Conference on Distributed Computing Systems, pp. 328-339, Atlanta, GA, USA, Jun. 2017.
[18] H. Li, K. Ota, and M. Dong, "Learning IoT in Edge: Deep Learning for the Internet of Things with Edge Computing," IEEE Network, vol. 32, no. 1, pp. 96- 101, Jan.-Feb. 2018.
[19] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, "Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge," Proceedings of the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 615-629, Xi'an, China, Apr. 2017.
[20] L. Li, K. Ota, and M. Dong, "Deep Learning for Smart Industry: Efficient Manufacture Inspection System with Fog Computing," IEEE Transactions on Industrial Informatics, vol. 14, no. 10, pp. 4665-4673, Oct. 2018.
[21] J. Chang, J. Kuo, C. Lin, W. Chen, and J. Sheu, "Ultra-Low-Latency Distributed Deep Neural Network over Hierarchical Mobile Networks," Proceedings of the IEEE Global Communications Conference, Waikoloa, HI, USA, Dec. 2019
[22] Z. Zhao, K. M. Barijough, and A. Gerstlauer, "DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters," IEEE Transactions on Computer-Aided Design of Integrated Circuits
and Systems, vol. 37, no. 11, pp. 2348-2359, Nov. 2018.
[23] J. Mao, X. Chen, K. W. Nixon, C. Krieger, and Y. Chen, "MoDNN: Local
Distributed Mobile Computing System for Deep Neural Network," Proceedings of the Design, Automation and Test in Europe Conference and Exhibition, pp. 1396-1401, Lausanne, Switzerland, Mar. 2017.
[24] R. Hadidi, J. Cao, M. Woodward, M. S. Ryoo, and H. Kim, "Musical Chair: Efficient Real-time Recognition using Collaborative IoT Devices," arXiv preprint arXiv:1802.02138, Feb. 2018.
[25] R. Hadidi, J. Cao, M. Woodward, M. S. Ryoo, and H. Kim, "Distributed Perception by Collaborative Robots," IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3709-3716, Oct. 2018.
[26] J. Mao, X. Chen, K. W. Nixon, C. Krieger, and Y. Chen, " MeDNN: A distributed mobile system with enhanced partition and deployment for large- scale DNNs," IEEE/ACM International Conference on Computer-Aided Design, pp. 751-756, Irvine, CA, Nov. 2017.
[27] J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," arXiv preprint arXiv: 1612.08242, Dec. 2016.
[28] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large- scale Image Recognition," Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA, May 2015.
[29] J. Dean, S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Proceedings of the 6th conference on Symposium on Operating Systems Design and Implementation, Vol. 6, San Francisco, CA, USA, Dec. 2004.
[30] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, "Pruning Convolutional Neural Networks for Resource Efficient Inference," arXiv preprint arXiv: 1611.06440, Nov. 2016
[31] A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew, “Deep Learning with COTS HPC Systems,” Proceedings of the 30th International Conference on International Conference on Machine Learning, Vol. 28, no. 3, pp. 1337–1345, Atlanta, GA, USA, June 2013.