圖形處理器(Graphics Processing Unit)是一種專門處理繪圖運算工作的處理器。因其強大的運算能力，所以也被廣泛的利用到許多領域。在雲端運算環境中，GPU虛擬化技術因其管理的方便性而日益重要。然而現行的GPU虛擬化技術多並非全虛擬化，需更改虛擬機所運行之驅動程式或程式函式庫，或是限定特定的硬體支援。GPUvm專案為一個支援Nvidia GPU的全虛擬化技術，但是只能在Xen hypervisor上運行。在這篇論文中，我們提出了KVM上的GPU全虛擬化技術GPUvmK。因為KVM和Xen hypervisor有基本上的差異，我們不僅僅是將GPUvm改到KVM上。GPUvmK主要更改的部分為aggregator, Qemu device model,memory management.由實驗可知，GPUvmK能到82%原生機器的執行效能。

Graphics processing Units (GPUs), which originally designed for the graphics calculating, have been widely adopted to general purpose computing in many domains owing to their massive computational power. In the era of cloud computing, GPU virtualization becomes an important technique for the better management of GPUs in data centers. However, most of current solutions are not full virtualization. They either need to modify the guest drivers or libraries, or restrict the hardware sharing capability. GPUvm is a full GPU virtualization solution for Nvidia GPUs, but it can only be executed on Xen hypervisors. In this thesis, we present a _rst full GPU virtualization solution on KVM (Kernel-based Virtual Machine), called GPUvmK. Our work is not merely a direct porting of GPUvm to KVM, since Xen and KVM have fundamental di_erences in their system architectures. Three major changes of GPUvmK are aggregator, QEMU device model, and memory management. Experiments show that GPUvmK can achieve 82% of native GPU performance. Comparing to GPUvm, GPUvmK has a larger memory initialization overhead but better execution performance.

1 Introduction 1
2 Background 6
2.1 GPU architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 PCI Con_guration . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Memory management . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.4 PCI Con_guration . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.5 Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.6 Memory management . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 GPUvm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Virtualization model . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.2 Aggregator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.3 Memory Isolation . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.4 Channel Isolation . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.5 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Implementation 15
3.1 Qemu-KVM Device Model . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Aggregator and KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4 Evaluation 21
4.1 Matrix addition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 Matrix multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3 Performance scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5 Conclusion 27

[1] EC Amazon. Amazon elastic compute cloud (amazon ec2). Amazon Elastic
Compute Cloud (Amazon EC2), 2010.
[2] I AMD. O virtualization technology (iommu) speci_cation. AMD Pub, 34434,
2007.
[3] Ardalan Amiri Sani, Kevin Boos, Shaopu Qin, and Lin Zhong. I/o paravirtu-
alization at the device _le boundary. In ACM SIGPLAN Notices, volume 49,
pages 319{332. ACM, 2014.
[4] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex
Ho, Rolf Neugebauer, Ian Pratt, and Andrew War_eld. Xen and the art of
virtualization. In ACM SIGOPS Operating Systems Review, volume 37, pages
164{177. ACM, 2003.
[5] Fabrice Bellard. Qemu, a fast and portable dynamic translator. In USENIX
Annual Technical Conference, FREENIX Track, pages 41{46, 2005.
[6] Micah Dowty and Jeremy Sugerman. Gpu virtualization on vmware's hosted
i/o architecture. ACM SIGOPS Operating Systems Review, 43(3):73{82, 2009.
[7] Jos_e Duato, Antonio J Pena, Federico Silla, Rafael Mayo, and Enrique S
Quintana-Ort__. rcuda: Reducing the number of gpu-based accelerators in
high performance clusters. In High Performance Computing and Simulation
(HPCS), 2010 International Conference on, pages 224{231. IEEE, 2010.
[8] Giulio Giunta, Ra_aele Montella, Giuseppe Agrillo, and Giuseppe Coviello.
A gpgpu transparent virtualization component for high performance comput-
ing clouds. In European Conference on Parallel Processing, pages 379{391.
Springer, 2010.
[9] Alex Herrera. Nvidia grid: Graphics accelerated vdi with the visual performance
of a workstation. Nvidia Corp, 2014.
[10] R Hiremane. Intel virtualization technology for directed i/o (intel vt-d). Tech-
nology@ Intel Magazine, 4(10), 2007.
[11] AMD Inc. AMD multiuser GPU.
[12] Shinpei Kato, Michael McThrow, Carlos Maltzahn, and Scott Brandt. Gdev:
First-class gpu resource management in the operating system. In Presented as
part of the 2012 USENIX Annual Technical Conference (USENIX ATC 12),
pages 401{412, 2012.
[13] David Kirk et al. Nvidia cuda software and gpu parallel computing architecture.
In ISMM, volume 7, pages 103{104, 2007.
[14] Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. kvm:
the linux virtual machine monitor. In Proceedings of the Linux symposium,
volume 1, pages 225{230, 2007.
[15] Patrick Kutch. Pci-sig sr-iov primer: An introduction to sr-iov technology. Intel
application note, pages 321211{002, 2011.
[16] H Andr_es Lagar-Cavilla, Niraj Tolia, Mahadev Satyanarayanan, and Eyal
De Lara. Vmm-independent graphics acceleration. In Proceedings of the 3rd in-
ternational conference on Virtual execution environments, pages 33{43. ACM,
2007.
[17] Antonios Motakis, Alvise Rigo, and Daniel Raho. Platform device assignment to
kvm-on-arm virtual machines via v_o. In Embedded and Ubiquitous Computing
(EUC), 2014 12th IEEE International Conference on, pages 170{177. IEEE,
2014.
[18] CUDA Nvidia. Programming guide, 2008.
[19] NVIDIA GRID NVIDIA and GRID GPUs. grid boards, nvidia corporation,
2013.
[20] Lin Shi, Hao Chen, Jianhua Sun, and Kenli Li. vcuda: Gpu-accelerated high-
performance computing in virtual machines. IEEE Transactions on Computers,
61(6):804{816, 2012.
[21] Yusuke Suzuki, Shinpei Kato, Hiroshi Yamada, and Kenji Kono. Gpuvm: Why
not virtualizing gpus at the hypervisor? In 2014 USENIX Annual Technical
Conference (USENIX ATC 14), pages 109{120, Philadelphia, PA, June 2014.
USENIX Association.
[22] Kun Tian, Yaozu Dong, and David Cowperthwaite. A full gpu virtualization
solution with mediated pass-through. In 2014 USENIX Annual Technical Con-
ference (USENIX ATC 14), pages 121{132, 2014.
[23] Tsan-Rong Tien and Yi-Ping You. Enabling opencl support for gpgpu in kernel-
based virtual machine. Software: Practice and Experience, 44(5):483{510, 2014.
[24] Sandra Wienke, Paul Springer, Christian Terboven, and Dieter an Mey. Ope-
nacc_rst experiences with real-world applications. In European Conference on
Parallel Processing, pages 859{870. Springer, 2012.