現今，動態電壓調配機制(Dynamic Voltage Scaling)被廣泛地用於解決時下電路設計之電源耗電問題。決定每個運行時間下的最佳操作電壓需考量工作負荷量變異、製程變異以及環境因素變異。在晶片設計階段，上述變異議題的預測實屬困難，故過去有許多文獻提出基於增強式學習法的動態電壓調配機制。其中，電壓緩降法則能基於臨界機率容忍時序錯誤，進而達到電源節省。然而，並沒有動態電壓調配機制的研究同時考慮電壓緩降法則。在本論文中，我們提出結合「Q適應學習法(Q-Learning Algorithm)」的動態電壓緩降調配機制，透過關鍵電路路徑監控器和基於時序容錯率的電壓預測，同時考慮溫度、電壓以及工作負荷量的晶片變異議題，達到效能最佳化及電源節省的目的。為評估其機制之優良，我們比較了兩種決定型動態電壓調配機制：步進式機制、統計模型式機制。相較於此兩種傳統機制，實驗結果顯示，針對三種45奈米工業製程晶片設計，我們所提出的Q適應學習法動態電壓緩降調配機制(搭配0.01的時序容忍機率值)可分別節省至多83.9%、29.1%的電源耗損。此研究為首度探討在電壓緩降法則下，基於增強式學習法的電壓緩降調配機制。

Dynamic voltage scaling (DVS) has been widely used to suppress power consumption in modern designs. The decision of optimal operating voltage at runtime should consider the variations in workload, process as well as environment. As these variations are hard to predict accurately at design time, various reinforcement learning based DVS schemes have been proposed in the literature. However, none of them can be readily applied to designs with graceful degradation, where timing errors are allowed with bounded probability to trade for further power reduction. In this thesis, we propose a Q-learning based DVS scheme dedicated to the designs with graceful degradation. We compare it with two deterministic DVS schemes, i.e., a stepping based scheme and a statistical modeling based scheme. Experimental results on three 45nm industrial designs show that the proposed Q-learning based scheme can achieve up to 83.9% and 29.1% power reduction respectively with 0.01 timing error probability bound. This is the first in-depth work to explore reinforcement learning based DVS schemes for designs with graceful degradation.

List of Contents 6
List of Figures 7
List of Tables 8
Chapter 1 INTRODUCTION 9
Chapter 2 PRELIMINARIES 13
2.1 Critical Path Monitor 13
2.2 Graceful Degradation 14
Chapter 3 MOTIVATION AND PROBLEM FORMULATION 16
3.1 Motivation 16
3.2 Problem Formulation 18
Chapter 4 Q-LEARNING BASED DVS SCHEME 21
4.1 Framework 21
4.2 Q-learning 22
Chapter 5 EXPERIMENTAL RESULTS 28
Chapter 6 CONCLUSIONS 35
ACKNOWLEDGEMENT 36
REFERENCES 37

[1] N. AbouGhazaleh, A. Ferreira, C. Rusu, R. Xu, F. Liberato, B. Childers, D. Mosse, R. Melhem, “Integrated CPU and l2 cache voltage scaling using machine learning,” in Proc. of conference on Languages, compilers, and tools for embedded systems, pp.41-50, 2007.
[2] T. Austin, V. Bertacco, D. Blaauw and T. Mudge, "Oppotunities and Challenges for Better Than Worst-Case Design", in Proc. Asia and South Pacific Design Automation Conf., 2005, pp. 2-7.
[3] T.D. Burd and R.W. Brodersen, “Design issues in dynamic voltage scaling,” in Proc. of the IEEE International Symposium on Low Power Electronics and Design, pp. 9–14, 2000.
[4] L. N. Chakrapani, B. E. S. Akgul, S. Cheemalavagu, P. Korkmaz, K. V. Palem and B. Seshasayee, “Ultra-Efficient (Embedded) SOC Architectures Based on Probabilistic CMOS (PCMOS) Technology”, in Proc. Design Automation and Test in Europe, 2006, pp. 1110-1115.
[5] Y.G. Chen, T.Wang, K.Y. Lai, W.Y. Wen, Y. Shi, and S.C. Chang, “Critical path monitor enabled dynamic voltage scaling for graceful degradation in sub-threshold designs,” in Proc. of 51st ACM/EDAC/IEEE Design Automation Conference (DAC), pp.1-6, June 2014.
[6] D. Chinnery, “High performance and low power design techniques for ASIC and custom in nanometer technologies,” in Proc. of the 2013 ACM international symposium on International Symposium on Physical Design (ISPD), pp.25-32. 2013.
[7] G. Dhiman and T. S. Rosing, “Dynamic voltage frequency scaling for multi-tasking systems using online learning,” in Proc. of 2007 ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED), pp.207-212, Aug. 2007.
[8] A. Drake, R. Senger, H. Deogun, G. Carpenter, S. Ghiasi, T. Nguyen, N. James, M. Floyd, and V. Pokala, “A Distributed Critical-Path Timing Monitor for a 65nm High-Performance Microprocessor,” in IEEE International Solid- State Circuits Conference (ISSCC), pp.398–399, 2007.
[9] F. Farahnakian, M. Ebrahimi, M. Daneshtalab, J. Plosila and P. Liljeberg, “Optimized Q-learning Model for Distributing Traffic in On-Chip Networks”, in Proc. IEEE International Conference on Networked Embedded Systems for Every Application (NESEA), 2012.
[10] M.E. Gomez, V. Santonja, "Self-similarity in I/O workload: analysis
and modeling", in Proc. of Workload Characterization: Methodology and Case Studies, pp.97 - 104, 1999.
[11] M.S. Gupta, K.K. Rangan, M.D. Smith, G.Y. Wei, and D. Brooks, “DeCoR: A Delayed Commit and Rollback mechanism for handling inductive noise in processors,” in Proc. of IEEE International Symposium on High Performance Computer Architecture (HPCA), pp.381–392, Feb 2008.
[12] R. Hegde and N. R. Shanbhag, “Energy-Efficient Signal Processing via Algorithmic Noise-Tolerance”, in Proc. International Symposium on Low Power Electronics and Design (ISLPED), 1999, pp. 30–35.
[13] R. Jejurikar, C. Pereira, and R. Gupta, “Leakage aware dynamic voltage scaling for real-time embedded systems,” in Proc. of 41st annual Design Automation Conference (DAC), pp.275-280.
[14] D. C. Juan, D. Marculescu, “Power-aware performance increase via core/uncore reinforcement control for chip-multiprocessors,” in Proc. of 2012 ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED), pp.97-102, Aug. 2012.
[15] D. C. Juan, S. Garg, J. Park, D. Marculescu, "Learning the optimal operating point for many-core systems with extended range voltage/frequency scaling," in Proc. of the Ninth IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis, pp.01-10.
[16] H. Jung and M. Pedram, “Supervised learning based power management for multicore processors,” in Proc. of IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 29, Issue 9, pp.1395-1408, Sep. 2010.
[17] A.B. Kahng, S. Kang, R. Kumar, and J. Sartori, “Slack redistribution for graceful degradation under voltage overscaling,” in Proc. of Asia and South Pacific Design Automation Conference (ASP-DAC), pp.852–831, Jan. 2012.
[18] J.T. Kao, M. Miyazaki, A.P. Chandrakasan, “A 175-mV multiply- accumulate unit using an adaptive supply voltage and body bias architecture,” in IEEE Journal of Solid-State Circuits, Vol.37, pp.1545–1554, 2002.
[19] M. Keating, D. Flynn, R. Aitken, A. Gibbons, and K. Shi, Low Power Methodology Manual, For System-on-Chip Design, Springer, 2007.
[20] C.R. Lefurgy, A.J. Drake, M.S. Floyd, M.S. Allen-Ware, B. Brock, J.A. Tierno, and J.B. Carter, “Active management of timing guardband to save energy in POWER7,” in Proc. of the 44th Annual IEEE/ACM International Symposium on Microarchitecture, pp.1–11, 2011.
[21] J. M. Levine, E. Stott and P. Y. K. Cheung, “Dynamic Voltage & Frequency Scaling with Online Slack Measurement,” in Proc. of ACM/SIGDA international symposuium on Fild-Programmable Gate Arrays, 2014.
[22] P. Pillai and K. G. Shin, “Real-time dynamic voltage scaling for low-power embedded operating systems,” in Proc. of eighteenth ACM symposium on Operating systems principles, pp.89-102, 2001.
[23] J. Pouwelse, K. Langendoen, and H. Sips, “Dynamic voltage scaling on a low-power microprocessor,” in Proc. of the 7th annual international conference on Mobile computing and networking, PP.251-259.
[24] C.A. Rummery and M. Niranjan, “On-Line Q-Learning Using Connectionist System”, in Technical Report CUED/F-INFENG/TR 166, Cambridge University, Cambridge, UK.
[25] H. Shen, J. Lu and Q. Qiu, “Learning based DVFS for simultaneous temperature, performance and energy management”, in Proc. of 13th International Symposium on Quality Electronic Design (ISQED), pp.747-754, March 2012.
[26] T. Simunic, L. Benini, A. Acquaviva, P. Glynn, and G. D. Micheli, “Dynamic voltage scaling and power management for portable systems,” in Proc. of the 38th annual Design Automation Conference, pp.254-529.
[27] R.S. Sutton and A.G. Barto, “Reinforcement Learning. An Introduction”, MIT Press, Cambridge, MA, 2000.
[28] J. N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning, 16(3):185–202, 1994.
[29] F. Vandeputte and L. Eeckhout, “Finding Stress Patterns in Microprocessor Workloads,” in Proc. of 4th International Conference on High Performance Embedded Architectures and Compilers, pp.153-167, 2009
[30] C. J. C. H. Watkins and P. Dayan, “Q-Learning”, in Proc. Machine Learning, pp.279-292, 1992.
[31] L. Yuan and G. Qu, “Analysis of energy reduction on dynamic voltage scaling-enabled systems,” in Proc. of IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 24, pp.1827–1837,Dec.2005