在智慧家庭中，為了達到較佳的能源使用率，電廠提供需量反應計畫給使用者，使用者透過改變原本用電習慣，降低電能花費，然而，需量反應計畫必須考量使用者偏好因素。因此，我們提出多目標強化學習方法設計一個基於用戶偏好之需量反應計畫，多目標強化學習方法改善了傳統的演算法，透過使用兩個學習表分別考慮花費及不滿意度，當使用者偏好改變時，多目標強化學習方法可以使用學習後的經驗，不用再次訓練就能達成收斂，此外，本方法應用的智慧家庭環境比起過往文獻更為完整，包含儲能裝置及綠能裝置，且考量到所有家電分類，符合更多家庭環境設定。最後，實驗利用真實資料證明，當使用者改變偏好，多目標強化學習方法的快速收斂，並且在考慮不確定性情況下能夠比過往方法有更好的表現。

In a smart home, a well-designed demand response (DR) program is essential for users to optimize energy usage under user preference. In this study, we propose a multiobjective reinforcement learning (MORL) algorithm to design a DR program. This approach improves conventional algorithms by mitigating the effect of change of user preference and can address uncertainty in future price. Through the use of two Q-tables, our algorithm jointly considers electricity cost and user dissatisfaction. When the user preference changes, the proposed MORL algorithm can use the previous experience to customize appliances’ scheduling and approach optimality fast. The proposed algorithm can be implemented in a smart home having an energy storage system, renewable energy source, and various types of appliances such as inflexible, time-flexible, and power-flexible ones. Numerical analysis using real-world data shows that the proposed approach can converge fast after the change of user preference and outperforms existing algorithms considering uncertainty in terms of electricity cost and user dissatisfaction.

摘要.....i
Abstract.....ii
Acknowledgement.....iii
Contents.....iv
List of Figures.....v
List of Tables.....vi
I. Introduction.....1
II. Related Work.....6
III. System Models.....8
3.1 Inflexible Appliance.....9
3.2 Time-Flexible Appliance.....9
3.3 Power-Flexible Appliance.....10
3.4 Energy Storage System.....11
3.5 Problem Formulation.....13
IV. User Preference Based Multi Objective Reinforcement Learning Approach.....15
V. Simulation Results.....21
VI. Conclusion.....31
Reference.....32

[1] H. Shareef, M. S. Ahmed, A. Mohamed, and E. Al Hassan, “Review on home energy management system considering demand responses, smart technologies, and intelligent controllers,” IEEE Access, vol. 6, pp. 24 498–24 509, May 2018.
[2] J. S. Vardakas, N. Zorba, and C. V. Verikoukis, “A survey on demand response programs in smart grids: Pricing methods and optimization algorithms,” IEEE Commun. Surveys Tuts., vol. 17, no. 1, pp. 152–178, Jan. 2015.
[3] A. Etxegarai, A. Bereziartua, J. A. Dañobeitia, O. Abarrategi, and G. Saldaña, “Impact of price based demand response programs for residential customers,” in Proc. IEEE Mediterranean Electrotechnical Conf., Marrakesh, Morocco, May 2018, pp. 204–208.
[4] N. G. Paterakis, O. Erdinç, A. G. Bakirtzis, and J. P. S. Catalão, “Optimal household appliances scheduling under day-ahead pricing and load-shaping demand response strategies,” IEEE Trans. Ind. Informat., vol. 11, no. 6, pp. 1509–1519, Dec. 2015.
[5] K. McKenna and A. Keane, “Residential load modeling of price based demand response for network impact studies,” IEEE Trans. Smart Grid, vol. 7, no. 5, pp. 2285–2294, Sep. 2016.
[6] S. Kim and G. B. Giannakis, “An online convex optimization approach to real time energy pricing for demand response,” IEEE Trans. Smart Grid, vol. 8, no. 6, pp. 2784–2793, Nov. 2017.
[7] Z. Zhang, F. Li, and H. Shi, “A pricing strategy reflecting the cost of power volatility to facilitate decentralized demand response,” IEEE Access, vol. 7, pp. 105 863–105 871, Aug. 2019.
[8] Z. Zhao, W. C. Lee, Y. Shin, and K. Song, “An optimal power scheduling method for demand response in home energy management system,” IEEE Trans. Smart Grid, vol. 4, no. 3, pp. 1391–1400, Sep. 2013.
[9] M. ShafieKhah and P. Siano, “A stochastic home energy management system considering satisfaction cost and response fatigue,” IEEE Trans. Ind. Informat., vol. 14, no. 2, pp. 629–638, Feb. 2018.
[10] S. Althaher, P. Mancarella, and J. Mutale, “Automated demand response from home energy management system under dynamic pricing and power and comfort constraints,” IEEE Trans. Smart Grid, vol. 6, no. 4, pp. 1874–1883, Jul. 2015.
[11] A. Heidari, Z. Y. Dong, D. Zhang, P. Siano, and J. Aghaei, “Mixed integer nonlinear programming formulation for distribution networks reliability optimization,” IEEE Trans. Ind. Informat., vol. 14, no. 5, pp. 1952–1961, May 2018.
[12] M. Alipour, K. Zare, and M. Abapour, “Minlp probabilistic scheduling model for demand response programs integrated energy hubs,” IEEE Trans. Ind. Informat., vol. 14, no. 1, pp. 79–88, Jan. 2018.
[13] S. Moon and J. Lee, “Multi-residential demand response scheduling with multi-class appliances in smart grid,” IEEE Trans. Smart Grid, vol. 9, no. 4, pp. 2518–2528, Jul. 2018.
[14] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean field multi-agent reinforcement learning,” in Proc. 35th ICML, vol. 80, Stockholmsmässan, Stockholm Sweden, Jul. 2018, pp. 5571–5580.
[15] E. C. Kara, M. Berges, B. Krogh, and S. Kar, “Using smart devices for system-level management and control in the smart grid: A reinforcement learning framework,” in Proc. IEEE Int. Conf. on Smart Grid Commun., Tainan, Taiwan, Nov. 2012, pp. 85–90.
[16] F. Ruelens, B. J. Claessens, S. Vandael, B. De Schutter, R. Babuška, and R. Belmans, “Residential demand response of thermostatically controlled loads using batch reinforcement learning,” IEEE Trans. Smart Grid, vol. 8, no. 5, pp. 2149–2159, Sep. 2017.
[17] Z. Wen, D. O’Neill, and H. Maei, “Optimal demand response using device-based reinforcement learning,” IEEE Trans. Smart Grid, vol. 6, no. 5, pp. 2312–2324, Sep. 2015.
[18] R. Lu, S. H. Hong, and M. Yu, “Demand response for home energy management using reinforcement learning and artificial neural network,” IEEE Trans. Smart Grid, vol. 10, no. 6, pp. 6629–6639, Nov. 2019.
[19] X. Xu, Y. Jia, Y. Xu, Z. Xu, S. Chai, and C. S. Lai, “A multi-agent reinforcement learning-based data-driven method for home energy management,” IEEE Trans. Smart Grid, vol. 11, no. 4, pp. 3201–3211, Jul. 2020.
[20] L. Park, Y. Jang, S. Cho, and J. Kim, “Residential demand response for renewable energy resources in smart grid systems,” IEEE Trans. Ind. Informat., vol. 13, no. 6, pp. 3165–3173, Dec. 2017.
[21] K. Li, P. Zhang, G. Li, F. Wang, Z. Mi, and H. Chen, “Day-ahead optimal joint scheduling model of electric and natural gas appliances for home integrated energy management,” IEEE Access, vol. 7, pp. 133 628–133 640, Sep. 2019.
[22] V. Pilloni, A. Floris, A. Meloni, and L. Atzori, “Smart home energy management including renewable sources: A qoe-driven approach,” IEEE Trans. Smart Grid, vol. 9, no. 3, pp. 2006–2018, May 2018.
[23] Y. Liu and S. Hu, “Renewable energy pricing driven scheduling in distributed smart community systems,” IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 5, pp. 1445–1456, May 2017.
[24] W. Yi, Y. Zhang, Z. Zhao, and Y. Huang, “Multiobjective robust scheduling for smart distribution grids: Considering renewable energy and demand response uncertainty,” IEEE Access, vol. 6, pp. 45 715–45 724, Sep. 2018.
[25] J. Shi, W. Lee, Y. Liu, Y. Yang, and P. Wang, “Forecasting power output of photovoltaic systems based on weather classification and support vector machines,” IEEE Trans. Ind. Appl., vol. 48, no. 3, pp. 1064–1069, Jun. 2012.
[26] H. Sangrody, M. Sarailoo, N. Zhou, N. Tran, M. Motalleb, and E. Foruzan, “Weather forecasting error in solar energy forecasting,” IET Renewable Power Generation, vol. 11, no. 10, pp. 1274–1280, Jul. 2017.
[27] L. An, J. Duan, M. Chow, and A. DuelHallen, “A distributed and resilient bargaining game for weather-predictive microgrid energy cooperation,” IEEE Trans. Ind. Informat., vol. 15, no. 8, pp. 4721–4730, Aug. 2019.
[28] D. Cerrai, D. W. Wanik, M. A. E. Bhuiyan, X. Zhang, J. Yang, M. E. B. Frediani, and E. N. Anagnostou, “Predicting storm outages through new representations of weather and vegetation,” IEEE Access, vol. 7, pp. 29 639–29 654, Mar. 2019.
[29] D. M. Rosewater, D. A. Copp, T. A. Nguyen, R. H. Byrne, and S. Santoso, “Battery energy storage models for optimal control,” IEEE Access, vol. 7, pp. 178 357–178 391, Dec. 2019.
[30] M. T. Lawder, B. Suthar, P. W. C. Northrop, S. De, C. M. Hoff, O. Leitermann, M. L. Crow, S. Santhanagopalan, and V. R. Subramanian, “Battery energy storage system (bess) and battery management system (bms) for grid-scale applications,” Proceedings of the IEEE, vol. 102, no. 6, pp. 1014–1030, Jun. 2014.
[31] X. Liang, “Emerging power quality challenges due to integration of renewable energy sources,” IEEE Trans. Ind. Appl., vol. 53, no. 2, pp. 855–866, 2017.
[32] Y. Wang, X. Lin, and M. Pedram, “Adaptive control for energy storage systems in households with photovoltaic modules,” IEEE Trans. Smart Grid, vol. 5, no. 2, pp. 992–1001, Mar. 2014.
[33] H. Chitsaz, P. Zamani-Dehkordi, H. Zareipour, and P. P. Parikh, “Electricity price forecasting for operational scheduling of behind-the-meter storage systems,” IEEE Trans. Smart Grid, vol. 9, no. 6, pp. 6612–6622, Nov. 2018.
[34] M. Ross, C. Abbey, F. Bouffard, and G. Joós, “Microgrid economic dispatch with energy storage systems,” IEEE Trans. Smart Grid, vol. 9, no. 4, pp. 3039–3047, Jul. 2018.
[35] Y. Wang, X. Lin, and M. Pedram, “A near-optimal model-based control algorithm for households equipped with residential photovoltaic power generation and energy storage systems,” IEEE Trans. Sustain. Energy, vol. 7, no. 1, pp. 77–86, Jan. 2016.
[36] B. Ke, T. Ku, Y. Ke, C. Chuang, and H. Chen, “Sizing the battery energy storage system on a university campus with prediction of load and photovoltaic generation,” IEEE Trans. Ind. Appl., vol. 52, no. 2, pp. 1136–1147, Apr. 2016.
[37] A. Fukushima, T. Yano, S. Imahara, H. Aisu, Y. Shimokawa, and Y. Shibata, “Prediction of energy consumption for new electric vehicle models by machine learning,” IET Intelligent Transport Systems, vol. 12, no. 9, pp. 1174–1180, Aug. 2018.
[38] Z. D. Asher, D. A. Baker, and T. H. Bradley, “Prediction error applied to hybrid electric vehicle optimal fuel economy,” IEEE Trans. Control Syst. Technol., vol. 26, no. 6, pp. 2121–2134, Nov. 2018.
[39] K. Mahmud, J. Ravishankar, M. J. Hossain, and Z. Y. Dong, “The impact of prediction errors in the domestic peak power demand management,” IEEE Trans. Ind. Informat., vol. 16, no. 7, pp. 4567–4579, Jul. 2020.
[40] M. Jia, F. Li, Y. Pu, and Z. Chen, “A lossless electrocardiogram compression system based on dual-mode prediction and error modeling,” IEEE Access, vol. 8, pp. 101 153–101 162, Jun. 2020.
[41] D. Kodaira, W. Jung, and S. Han, “Optimal energy storage system operation for peak reduction in a distribution network using a prediction interval,” IEEE Trans. Smart Grid, vol. 11, no. 3, pp. 2208–2217, May 2020.
[42] X. Lee, M. Yan, F. Y. Xu, Y. Wang, Y. Fan, Z. Lee, Y. Wen, M. Shahidehpour, and L. L. Lai, “Virtual storage-based dsm with error-driven prediction modulation for microgrids,” IEEE Access, vol. 7, pp. 71 109–71 118, Jun. 2019.
[43] R. Singh, R. C. Bansal, A. R. Singh, and R. Naidoo, “Multi-objective optimization of hybrid renewable energy system using reformed electric system cascade analysis for islanding and grid connected modes of operation,” IEEE Access, vol. 6, pp. 47 332–47 354, Sep. 2018.
[44] B. S. Sami, “Intelligent energy management for off-grid renewable hybrid system using multi-agent approach,” IEEE Access, vol. 8, pp. 8681–8696, Jan. 2020.
[45] C. Tsai, T. M. Beza, E. M. Molla, and C. Kuo, “Analysis and sizing of mini-grid hybrid renewable energy system for islands,” IEEE Access, vol. 8, pp. 70 013–70 029, Apr. 2020.
[46] Z. Yulong, W. Weida, X. Changle, l. Hui, and R. Langari, “Research and bench test of nonlinear model predictive control-based power allocation strategy for hybrid energy storage system,” IEEE Access, vol. 6, pp. 70 770–70 787, Dec. 2018.
[47] F. Alfaverh, M. Denaï, and Y. Sun, “Demand response strategy based on reinforcement learning and fuzzy reasoning for home energy management,” IEEE Access, vol. 8, pp. 39 310–39 321, Mar. 2020.
[48] K. Ma, T. Yao, J. Yang, and X. Guan, “Residential power scheduling for demand response in smart grid,” Int. J. Electr. Power Energy Syst., vol. 78, pp. 320–325, Jun. 2016.
[49] H. Chang, W. Chiu, H. Sun, and C. Chen, “User-centric multiobjective approach to privacy preservation and energy cost minimization in smart home,” IEEE Syst. J., vol. 13, no. 1, pp. 1030–1041, Mar. 2019.
[50] F. Luo, W. Kong, G. Ranzi, and Z. Y. Dong, “Optimal home energy management system with demand charge tariff and appliance operational dependencies,” IEEE Trans. Smart Grid, vol. 11, no. 1, pp. 4–14, Jan. 2020.
[51] C. J. Watkins and P. Dayan, “Qlearning,” Machine learning, vol. 30, no. 4, pp. 279–292, 1992.
[52] X. Huang, S. H. Hong, and Y. Li, “Hour-ahead price based energy management scheme for industrial facilities,” IEEE Trans. Ind. Informat., vol. 13, no. 6, pp. 2886–2898, Dec. 2017.
[53] PJM interconnection. Accessed on Jan. 2, 2019. [Online]. Available: http://www.pjm.com
[54] F. Wang, Z. Mi, S. Su, and H. Zhao, “Short-term solar irradiance forecasting model based on artificial neural network using statistical feature parameters,” vol. 5, no. 5, pp. 1355–1370, May 2012.