InfiniBand 是一個高頻寬、低延遲的網路架構，它廣泛用於高效能運算領域，但在 QEMU/KVM 虛擬機上以軟體虛擬化進行，效能與記憶體分配瓶頸仍有待突破。InfiniBand 介面卡 (HCA) 本身即有「透通作業系統」 (OS-bypass) 的特性，該特性允許裝置不經由作業系統，直接與應用程式透過直接記憶體存取 (DMA) 溝通以達到更高效能。前人提出的 Virt-IB 雖然可以達成 QEMU/KVM 中使用 InfiniBand，但 Virt-IB 沒有利用 OS-bypass 的特性，因此只能達成原始效能的 25%～50%。

本論文提出改進的 InfiniBand 虛擬化，稱為 Virt-IB-Memlink。我們以 Virt-IB 為基礎，加上兩個主要的方法來達到更高效能：第一個方法將虛擬機器內的記憶體連續化，以方便硬體直接存取虛擬機器內的記憶體；第二個方法使得虛擬機器可以直接通知裝置，不透過虛擬機監視器 (VMM) 與作業系統干涉以提升效能。透過上述方式，我們的 InfiniBand 虛擬化系統雖然會影響記憶體隨機讀取效率，但可以在一次傳輸較大資料下達成原生效能。

InfiniBand is a high-bandwidth, low-latency interconnection architecture, and is commonly used in clusters. However, there are performance overhead and memory allocation problem using InfiniBand in QEMU/KVM under software-based virtualization. InfiniBand host channel adapters have OS-bypass characteristic, which device communicates with user-space applications without OS involvement or data copying, thus provides better performance. Virt-IB is an InfiniBand virtualization implementation on QEMU/KVM. It was implemented without benefiting from OS-bypass, therefore it provides only 25%–50% of native throughput.

In this work, we provide Virt-IB-Memlink, an improved InfiniBand virtualization. Our work is based on Virt-IB, with two major improvements to achieve higher performance. One is memlink, which ensures memory contiguity and brings zero-copy between guest and host. The other is doorbell mapping, which guest user-space can notify the device directly by assigning the doorbell memory page onto guest memory space. With these two enhancements, in spite of performance degradation of random memory read access, our InfiniBand virtualization delivers native performance when transmitting large chunk of data at once.

1 Introduction

2 Related Work
2.1 QEMU/KVM
2.1.1 Virtio
2.2 InfiniBand Architecture
2.3 OS- and VMM-Bypass I/O
2.3.1 OS-Bypass I/O
2.3.2 VMM-Bypass I/O
2.4 Virt-IB
2.4.1 Sending and Receiving Implementation

3 Design and Implementation
3.1 memlink: Memory Continuity and Zero-Copy Mechanism
3.1.1 Usages in InfiniBand Virtualization
3.2 Doorbell Mapping
3.3 Other Implementation Details in Virt-IB-Memlink
3.3.1 Handling Device File Openings
3.3.2 Completion Queue Polling Using Pend Paradigm

4 Performance Evaluation
4.1 Configurations
4.2 memlink Memory Access Benchmark
4.3 ibv_rc_pingpong Benchmarks
4.4 MPI Benchmarks
4.4.1 OSU Micro-Benchmarks
4.4.2 HPCC PTRANS Benchmark

5 Conclusions and Future Work

[1] Charles David Graziano. “A performance analysis of Xen and KVM hypervisors for hosting the Xen Worlds Project” (2011).
[2] Fabrice Bellard. “QEMU, a Fast and Portable Dynamic Translator.” USENIX Annual Technical Conference, FREENIX Track. 2005, pp. 41–46.
[3] Avi Kivity et al. “kvm: the Linux virtual machine monitor”. Proceedings of the Linux Symposium. Vol. 1. 2007, pp. 225–230.
[4] Gregory F Pfister. “An introduction to the InfiniBand architecture”. High Performance Mass Storage and Parallel I/O 42 (2001), pp. 617–632.
[5] TOP500 Supercomputer Sites. Nov. 2013. URL : http://www.top500.org/lists/2013/11/ .
[6] Paul Grun. Introduction to InfiniBand TM for End Users. White Paper. 2010. URL : https://cw.infinibandta.org/document/dl/7268 .
[7] Raoul AF Bhoedjang, Tim Ruhl, and Henri E Bal. “User-level network interface protocols”. Computer 31.11 (1998), pp. 53–60.
[8] Jiuxing Liu et al. “High Performance VMM-Bypass I/O in Virtual Machines.”
[9] Patrick Kutch. “PCI-SIG SR-IOV Primer: An Introduction to SR-IOV Technology”. Application note (2011).
[10] Yi-Man Ma, Che-Rung Lee, and Yeh-Ching Chung. “InfiniBand virtualization on KVM”. Cloud Computing Technology and Science (CloudCom), 2012 IEEE 4th International Conference on. IEEE. 2012, pp. 777–781.
[11] Matt Gillespie. “Best Practices for Paravirtualization Enhancements from Intel virtualization technology: EPT and VT-d”. Retrieved November 26 (2009), p. 2013.
[12] Rusty Russell. “virtio: towards a de-facto standard for virtual I/O devices”. ACM SIGOPS Operating Systems Review 42.5 (2008), pp. 95–103.
[13] RDMA Aware Networks Programming User Manual. Rev. 1.6. Mellanox Technologies. 2014.
[14] Jiuxing Liu et al. “Microbenchmark performance comparison of high-speed cluster interconnects”. Micro, IEEE 24.1 (2004), pp. 42–51.
[15] Sayantan Sur et al. “RDMA read based rendezvous protocol for MPI over InfiniBand: design alternatives and benefits”. Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programming. ACM. 2006, pp. 32–39.
[16] Piotr R Luszczek et al. “The HPC Challenge (HPCC) Benchmark Suite”. Proceedings of the 2006 ACM/IEEE Conference on Supercomputing. SC ’06. Tampa, Florida:ACM, 2006. ISBN : 0-7695-2700-0. DOI : 10.1145/1188455.1188677 . URL : http://doi.acm.org/10.1145/1188455.1188677 .