本論文旨在提出考量電池放電深度及壽命的學習式能源管理策略，其策略會考量電池放電深度與電池壽命之間的抉擇。過去因為缺乏電池放電深度對應電池耗損邊際成本的模型，通常會忽略電池放電深度及電池壽命對能源管理策略的影響。此篇論文中，我們提出一個新穎的電池成本計算方法，其考慮到每次電池使用的放電深度，此方法將會用於設計使用強化學習及線性值函數近似法之日前能源管理策略。此能遠管理策略將決定要從日前市場購買多少能源供給下一日使用。此論文提出之能源管理系統將使用最小平方策略迭代法及次狀態策略迭代法學習能源管理策略，最小平方策略是用於值函數的線性近似，而次狀態策略迭代法是用於下一天系統參數的預測。此論文的模擬是基於實際的非商用單一家庭負載、日前電價、即時電價及再生能源模擬器。考慮電池充放電的代價更能提供能源管理系統精確的整體系統花費，使能源管理策略能同時顧慮購電成本及電池壽命成本。

This work proposes a learning-based energy management policy that takes into consideration the trade-off between the depth-of-discharge (DoD) and the lifetime of batteries. The impact of DoD on the energy management policy is often neglected in the past due to the inability to model its effect on the marginal cost per battery usage. In this work, a novel battery cost evaluation method that takes into consideration the DoD of each battery usage is proposed, and is utilized to devise the day-ahead energy management policy using reinforcement learning and linear value-function approximations. The policy determines the amount of energy to purchase for the next day in the day ahead market. A least-square policy iteration (LSPI) with linear approximations of the value function is used to learn the energy management policy. Simulations are provided based on real load profiles, pricing data, and renewable energy arrival statistics. The consideration of the battery cost due to DoD provides a more accurate evaluation of the actual energy cost and leads to an improved energy management policy.

Abstract i
Contents ii
List of Figures iv
List of Algorithms v
1 Introduction 1
2 Brief Review of Reinforcement Learning and Least-Square Policy Iteration 5
2.1 Markov Decision Process 5
2.2 Least-Square Policy Iteration 6
3 System Model and Problem Formulation 12
3.1 Cost of Energy Purchase 13
3.2 Cost of Battery Usage 14
4 Learning-Based Energy Management Policy with DoD Considerations 18
4.1 State Transition 19
4.2 Basis 20
4.3 Real-Life Data Prediction 21
5 Simulation 27
6 Conclusion 37
Bibliography 38

[1] A. Ipakchi and F. Albuyeh. Grid of the future. IEEE Power and Energy Magazine, 7(2):52–62, 2009.
[2] Xi Fang, Satyajayant Misra, Guoliang Xue, and Dejun Yang. Smart grid - the new and improved power grid: A survey. IEEE Communications Surveys Tutorials, 14(4):944–980, 2012.
[3] Chris S King. The economics of real-time and time-of-use pricing for residential consumers. American Energy Institute, Tech. Rep, 2001.
[4] Xiaodao Chen, Tongquan Wei, and Shiyan Hu. Uncertainty-aware household appliance scheduling considering dynamic electricity pricing in smart home. IEEE Transactions on Smart Grid, 4(2):932–941, 2013.
[5] Hunt Allcott. Rethinking real-time electricity pricing. Resource and Energy Economics, 33(4):820 – 842, 2011.
[6] S.D. Smitha, J.S. Savier, and F.M. Chacko. Intelligent control system for efficient energy management in commercial buildings. In Proc. Annual International Conference on Communications and Renewable Energy, pages 1–6, June 2013.
[7] Jinsoo Han, Chang sic Choi, Wan-Ki Park, Ilwoo Lee, and Sang-Ha Kim. Smart home energy management system including renewable energy based on zigbee and plc. IEEE Transactions on Consumer Electronics, 60(2):198–202, May 2014.
[8] Sungjin Lee, Beom Kwon, and Sanghoon Lee. Joint energy management system of electric supply and demand in houses and buildings. IEEE Transactions on Power Systems, 29(6):2804–2812, Nov 2014.
[9] Li Ping Qian, Y.J.A. Zhang, Jianwei Huang, and Yuan Wu. Demand response management via real-time electricity price control in smart grids. IEEE Journal on Selected Areas in Communications, 31(7):1268–1280, July 2013.
[10] L. Jiang and S. Low. Multi-period optimal energy procurement and demand response in smart grid with uncertain supply. In Porc. IEEE Conference on Decision and Control and European Control Conference (CDC-ECC), pages 4348–4353, 2011.
[11] I. Atzeni, L.G. Ordonez, G. Scutari, D.P. Palomar, and J.R. Fonollosa. Demand-side management via distributed energy generation and storage optimization. IEEE Transactions on Smart Grid, 4(2):866–876, 2013.
[12] I. Koutsopoulos, V. Hatzi, and L. Tassiulas. Optimal energy storage control policies for the smart power grid. In Proc. IEEE International Conference on Smart Grid Communications (SmartGridComm), pages 475–480, 2011.
[13] T. Erseghe, A. Zanella, and C.G. Codemo. Optimal and compact control policies for energy storage units with single and multiple batteries. IEEE Transactions on Smart Grid, 5(3):1308–1317, 2014.
[14] D.K. Maly and K.S. Kwan. Optimal battery energy storage system (bess) charge scheduling with dynamic programming. IEE Proceedings Science, Measurement and Technology, 142(6):453–458, 1995.
[15] P.M. van de Ven, N. Hegde, L. Massoulie, and T. Salonidis. Optimal control of end-user energy storage. IEEE Transactions on Smart Grid, 4(2):789–797, 2013.
[16] Li Xin, Zang Chuanzhi, Zeng Peng, and Yu Haibin. Genetic based fuzzy q-learning energy management for smart grid. In Proc. Chinese Control Conference (CCC), pages 6924–6927, July 2012.
[17] Lauren Hannah and David B Dunson. Approximate dynamic programming for storage problems. In Proc. International Conference on Machine Learning, pages 337–344, 2011.
[18] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1107–1149, 2003.
[19] David Linden and Thomas B Reddy. Handbook of batteries, 1865.
[20] David Dallinger. Plug-in electric vehicles integrating fluctuating renewable electricity, volume 20. Kassel University Press, 2013.
[21] Vivek Ramavajjala and Charles Elkan. Policy iteration based on a learned transition model. In PeterA. Flach, Tijl De Bie, and Nello Cristianini, editors, Machine Learning and Knowledge Discovery in Databases, volume 7524 of Lecture Notes in Computer Science, pages 211–226. Springer Berlin Heidelberg, 2012.
[22] ComEd. http://www.comed.com/. Accessed: 2015-05-10.
[23] NREL. http://pvwatts.nrel.gov. Accessed: 2015-05-10.