目前，並行計算機和GPGPU等加速器通常用於大規模計算。
然而，這些架構是通用的結構，處理器核的硬件很大，可以安裝在芯片中核的數量，以及可以連接到共享高速緩存的核的數量是有限的。

因此，在縮放時獲取共享數據的延遲問題有嚴重化趨勢。
硬件利用的效率和功耗性能方面，需要進一步提高效率。

這項研究中，我專注於EXACC（Extreme SIMD Accelerator），它被提出作為加速器的結構，目標是Stencil Computation，其中數據的計算量很大，並且相鄰之間的數據傳輸很重要。

我對這個加速器作出了評估，從結果中找到了架構的瓶頸，並以此作為基礎，改進了管道，令它能夠在3D Himeno基準測試中提高4.98％的執行效率，並且功耗降低了3.43％。

At present, supercomputers are developed to realize 1 EFLOPS. These accelerators such as parallel computers and GPGPU are often used for large-scale calculation.
However, these architectures have general-purpose structures. The hardware size of their processor cores is large. In addition, the number of cores that can be mounted in the chip, and the number of cores that can be connected to the shared cache are limited.
Therefore, that is a problem of latency in acquiring shared data increases when scaling. Due to these problems, the execution efficiency of the present architectures has been reduced. In terms of hardware utilization efficiency and power consumption performance, further efficiency improvement is required.

In this research, I focused on EXACC (Extreme SIMD Accelerator) which is proposed as a structure of accelerator targeted to stencil applications where computation amount for data is large and data transfer between adjacencies is important.
This architecture attains the extremely high execution efficiency by using neighbor core communication which is most important in stencil calculation.
I evaluated this accelerator, and found the bottleneck of the architecture from the analysis, and improved the pipeline based on the result. It was able to improve the execution efficiency performance of 4.98% on the 3D Himeno benchmark with the reduction of the power consumption by 3.43%, and the hardware area by 21.4%. The proposed improvement for the accelerator is suitable for the performance-energy-efficient applications on the supercomputers.

1 Introduction 1
2 Relayted work 3
2.1 PEZY-SC2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 NEC SX-Aurora TSUBASA . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Scalable Stencil-Computation Accelerator . . . . . . . . . . . . . . . . . . . . 6
2.4 GRAPE-DR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 Accelerator comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Architecture of original EXACC 11
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Communication interface with PC . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.1 Control of Master Processor . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.2 Input and output of data and programs . . . . . . . . . . . . . . . . . 13
3.2.3 Debug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Memory hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.1 GM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.2 BM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.3 LM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Master Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Instruction Sequencer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5.1 Global sequencer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.5.2 Local sequencer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 Data DMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7 PE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7.1 PE registers and memory . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.7.2 BM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.7.3 LM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.7.4 PE instruction set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.7.5 Conditional instruction . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.7.6 PE mesh network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.8 BB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.9 Step of execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4 PE Assembler 27
4.1 Basic format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Definition of register and memory . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3 Definition of program thread . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.4 Prefix instruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5 FPGA Implementation 32
5.1 Basic specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Performance evaluation target . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2.1 GM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2.2 Master Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.2.3 SD Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.2.4 Instruction Sequencer . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.2.5 PEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.3 Power evaluation target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6 Evaluation of original EXACC 36
6.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.1.1 2D Laplace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.1.2 3D Himeno Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.2 Evaluate target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.3 Result of 2D Laplace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.4 Result of 3D Himeno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.5.1 Bottleneck-1 : long PE pipeline stages . . . . . . . . . . . . . . . . . 42
6.5.2 Bottleneck-2 : concentration of access to LM . . . . . . . . . . . . . . 42
7 Architecture of improved EXACC 45
7.1 Improvement of pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
7.2 2 banks local memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.3 Discussion about improved architecture . . . . . . . . . . . . . . . . . . . . . 49
8 Evaluation of improved EXACC 50
8.1 2 types of evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.2 Execution efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.2.1 Type-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.2.2 Type-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
8.3 Hardware area and power consumption . . . . . . . . . . . . . . . . . . . . . 52
8.3.1 Critical path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.3.2 Reduction number of instructions . . . . . . . . . . . . . . . . . . . . 55
9 Conclusion 57

[1] TOP500 Supercomputer Sites. https://www.top500.org/.
[2] Green500 — TOP500 Supercomputer Sites. https://www.top500.org/green500/.
[3] George Chrysos. Intel⃝R xeon phi coprocessor-the architecture. Intel Whitepaper, Vol.
176, , 2014.
[4] Kentaro Sano, Yoshiaki Hatsuda, and Satoru Yamamoto. Multi-fpga accelerator for
scalable stencil computation with constant memory bandwidth. IEEE Transactions on
Parallel and Distributed Systems, Vol. 25, No. 3, pp. 695–705, 2014.
[5] 伊藤剛浩, 山口佳樹, 児玉祐悦, 山本淳二, 中川八穂子, 朴泰祐, 佐藤三久. D-6-9 fpga
を用いた extremesimd アーキテクチャの予備評価 (d-6. コンピュータシステム a (高性
能ハードウェア), 一般セッション). 電子情報通信学会総合大会講演論文集, Vol. 2015,
No. 1, p. 73, 2015.
[6] 伊藤剛浩, 山口佳樹, 朴泰祐, 佐藤三久, 児玉祐悦, 李珍泌, 山本淳二, 中川八穂子. 大規
模 simd アーキテクチャの提案とその検証: Fpga による予備評価 (リコンフィギャラブ
ルシステム). 電子情報通信学会技術研究報告= IEICE technical report: 信学技報, Vol.
116, No. 53, pp. 55–60, 2016.
[7] 宇川斉志, 佐藤三久, 朴泰祐, 児玉祐悦, 山口佳樹, 山本淳二ほか. Extreme simd アーキ
テクチャのプログラミングモデル拡張 c による性能評価. 研究報告ハイパフォーマンス
コンピューティング (HPC), Vol. 2015, No. 24, pp. 1–8, 2015.
[8] 「 演 算 加 速 機 構 を 持 つ 将 来 の HPCI シ ス テ ム に 関 す る 調 査 研 究 」中 間 報 告
. http://www.mext.go.jp/b_menu/shingi/chousa/shinkou/020/shiryo/__
icsFiles/afieldfile/2013/04/25/1333805_8.pdf.
[9] Junichiro Makino, Kei Hiraki, and Mary Inaba. Grape-dr: 2-pflops massively-parallel
computer with 512-core, 512-gflops processor chips for scientific computing. In Proceedings
of the 2007 ACM/IEEE conference on Supercomputing, p. 18. ACM, 2007.
[10] 鳥居淳, 石川仁, 木村耕行, 齊藤元章. グリーンスーパーコンピュータ zettascaler の技術
と今後の展望. 電子情報通信学会論文誌 C, Vol. 100, No. 11, pp. 537–544, 2017.
[11] Takuji Mitsuishi, Takahiro Kaneda, Sunao Torii, and Hideharu Amano. Implementing
breadth-first search on a compact supercomputer suiren. In Computing and Networking
(CANDAR), 2016 Fourth International Symposium on, pp. 395–401. IEEE, 2016.
[12] NEC SX-Aurora TSUBASA - Vector Engine. https://www.nec.com/en/global/
solutions/hpc/sx/vector_engine.html.
[13] Shintaro Momose, Takashi Hagiwara, Yoko Isobe, and Hiroshi Takahara. The brandnew
vector supercomputer, sx-ace. In International Supercomputing Conference, pp.
199–214. Springer, 2014.
[14] Hiroaki Kobayashi, Ryusuke Egawa, Hiroyuki Takizawa, Koki Okabe, Akihiko Musa,
Takashi Soga, and Yoichi Shimomura. First experiences with nec sx-9. In High Performance
Computing on Vector Systems 2008, pp. 3–11. Springer, 2009.
[15] Hongshin Jun, Jinhee Cho, Kangseol Lee, Ho-Young Son, Kwiwook Kim, Hanho Jin,
and Keith Kim. Hbm (high bandwidth memory) dram technology and architecture. In
Memory Workshop (IMW), 2017 IEEE International, pp. 1–4. IEEE, 2017.
[16] 演算加速機構を持つ将来の HPCI システムに関する調査研究. http://www.mext.go.
jp/b_menu/shingi/chousa/shinkou/028/shiryo/__icsFiles/afieldfile/2012/
08/14/1324574_3_1.pdf.
[17] Himeno benchmark. http://accc.riken.jp/en/supercom/himenobmt/.