近年來，由於能源貿易自由化與再生能源的普及，可再生能源發電場與能源用戶之再生能源貿易引起廣泛關注，且發展出新型能源市場。目前廣泛使用之市場架構為以聚合商為中心之貿易情境與端對端貿易。以聚合商為中心的貿易情境具有用戶隱私保護與高度溝通效率之特性，而端對端貿易能夠讓市場參與者自行決定價錢與數量，不需要依靠第三者幫助協調。本論文提出混和式再生能源交易情境，結合以聚合商為中心之貿易情境與端對端貿易之優點，並提出基於多代理Q 學習競價策略確保市場參與者能夠於能源貿易中獲取最大利益。基於本論文提出之混和式再生能源交易情境，市場參與者(再生能源發電廠及能源用戶) 首先提供發電量與需求量予聚合商，聚合商回傳總發電量與總需求量予市場參與者，接下來各個市場參與者採用本論文提出之基於多代理Q 學習競價策略進行報價，最終由聚合商基於報價進行市場參與者之能源分配。真實數值模擬結果說明本論文提出之方法於再生能源發電廠之總收益、能源成本、能源用戶之再生能源需求滿足度與價格彈性皆超越現有方法。

Owing to energy liberalization and increasing penetration of renewables, renewable energy trading among suppliers and users has gained much attention and created a new market. This study thus proposes a hybrid renewable energy trading scheme: it preserves
benefits of aggregate trading, such as privacy protection and effective maintenance of communications infrastructure; and it allows market participants to have control over bidding prices and energy quantities, an appealing feature in peer-to-peer trading. To ensure beneficial bidding for renewable generators and end users (EUs), a multiagent Q-learning (MAQL) based bidding strategy is developed to maximize their accumulated rewards. In our hybrid scheme, market participants first provide their information about renewable
supply and demand to an aggregator. The aggregator then returns the information about aggregate supply and demand. Renewable generators and EUs employ the proposed MAQL algorithm to bid for desired amounts of power to be sold and bought, respectively. Finally, the aggregator coordinates energy trading between renewable generators and EUs.
Numerical analysis using real-world data shows that the proposed approach outperforms comparable methods in terms of profits of renewable generators, energy costs and satisfaction level of meeting a basic renewable demand of EUs, and price elasticity.

摘要. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
Abstract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
II. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
III. System Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.1 Renewable Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 End Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Aggregator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
IV. Proposed Hybrid Bidding Scheme . . . . . . . . . . . . . . . . . . . 15
4.1 Desired Demand in Step 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Learning Algorithms in Step 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
V. Numerical Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
VI. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
A.1 Stepbystep
derivation of QLearning
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
A.2 Nonlinear profit model of aggregator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

[1] A. Z. Amin, “Renewable energy prospects for the European Union,” International Renewable
Energy Agency, Tech. Rep., Feb. 2018.
[2] Renewable energy in Taiwan. [Online]. Available: https://en.wikipedia.org/wiki/
Renewable_energy_in_Taiwan
[3] R. Green, “Electricity liberalisation in Europe–how competitive will it be?” Energy Policy,
vol. 34, no. 16, pp. 2532–2541, Nov. 2006.
[4] L. M. Rathke, “The effects of electricity market liberalisation in the European Union,” Master’s
thesis, University of Twente, 2015.
[5] S. Burger, J. P. ChavesÁvila,
C. Batlle, and I. J. PérezArriaga,
“A review of the value of
aggregators in electricity systems,” Renew. Sustain. Energy Rev., vol. 77, pp. 395–405, Sep.
2017.
[6] M. Pilz and L. AlFagih,
“Recent advances in local energy trading in the smart grid based on
gametheoretic
approaches,” IEEE Trans. Smart Grid, vol. 10, no. 2, pp. 1363–1371, Mar.
2019.
[7] M. M. Esfahani, A. Hariri, and O. A. Mohammed, “A multiagentbased
gametheoretic
and
optimization approach for market operation of multimicrogrid systems,” IEEE Trans. Ind.
Informat., vol. 15, no. 1, pp. 280–292, Jan. 2019.
[8] Y. Wang, W. Saad, Z. Han, H. V. Poor, and T. Başar, “A gametheoretic
approach to energy
trading in the smart grid,” IEEE Trans. Smart Grid, vol. 5, no. 3, pp. 1439–1450, May 2014.
[9] K. Zhang, Y. Mao, S. Leng, S. Maharjan, Y. Zhang, A. Vinel, and M. Jonsson, “Incentivedriven
energy trading in the smart grid,” IEEE Access, vol. 4, pp. 1243–1257, 2016.
[10] N. Rashedi, M. A. Tajeddini, and H. Kebriaei, “Markov game approach for multiagent
competitive
bidding strategies in electricity market,” IET Generation, Transmission & Distribution,
vol. 10, no. 15, pp. 3756–3763, Nov. 2016.
[11] T. Dai and W. Qiao, “Trading wind power in a competitive electricity market using stochastic
programing and game theory,” IEEE Trans. Sustain. Energy, vol. 4, no. 3, pp. 805–815, Mar.
2013.
[12] G. N. Bathurst, J. Weatherill, and G. Strbac, “Trading wind generation in short term energy
markets,” IEEE Trans. Power Syst., vol. 17, no. 3, pp. 782–789, Nov. 2002.
[13] J. Matevosyan and L. Soder, “Minimization of imbalance cost trading wind power on the
shortterm
power market,” IEEE Trans. Power Syst., vol. 21, no. 3, pp. 1396–1404, Jul.
2006.
[14] P. Pinson, C. Chevallier, and G. N. Kariniotakis, “Trading wind generation from shortterm
probabilistic forecasts of wind power,” IEEE Trans. Power Syst., vol. 22, no. 3, pp. 1148–
1156, Jul. 2007.
[15] C. Tham and T. Luo, “Sensingdriven
energy purchasing in smart grid cyberphysical
system,”
IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 4, pp. 773–784, Jul. 2013.
[16] S. Maharjan, Q. Zhu, Y. Zhang, S. Gjessing, and T. Basar, “Dependable demand response
management in the smart grid: A stackelberg game approach,” IEEE Trans. Smart Grid,
vol. 4, no. 1, pp. 120–132, Mar. 2013.
[17] G. E. Asimakopoulou, A. L. Dimeas, and N. D. Hatziargyriou, “Leaderfollower
strategies
for energy management of multimicrogrids,”
IEEE Trans. Smart Grid, vol. 4, no. 4, pp.
1909–1916, Dec. 2013.
[18] W. Tushar, J. A. Zhang, D. B. Smith, H. V. Poor, and S. Thiébaux, “Prioritizing consumers
in smart grid: A game theoretic approach,” IEEE Trans. Smart Grid, vol. 5, no. 3, pp. 1429–
1438, Apr. 2014.
[19] W. Wei, F. Liu, and S. Mei, “Energy pricing and dispatch for smart grid retailers under demand
response and market price uncertainty,” IEEE Trans. Smart Grid, vol. 6, no. 3, pp. 1364–1374,
Dec. 2015.
[20] E. Nekouei, T. Alpcan, and D. Chattopadhyay, “Gametheoretic
frameworks for demand response
in electricity markets,” IEEE Trans. Smart Grid, vol. 6, no. 2, pp. 748–758, Nov.
2015.
[21] T. Choi, X. Li, and C. Ma, “Searchbased
advertising auctions with choicebased
budget
constraint,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 45, no. 8, pp. 1178–1186, Aug.
2015.
[22] N. Li, L. Chen, and M. A. Dahleh, “Demand response using linear supply function bidding,”
IEEE Trans. Smart Grid, vol. 6, no. 4, pp. 1827–1838, Mar. 2015.
[23] M. N. Faqiry and S. Das, “Doublesided
energy auction in microgrid: Equilibrium under
price anticipation,” IEEE Access, vol. 4, pp. 3794–3805, Jul. 2016.
[24] D. An, Q. Yang, W. Yu, X. Yang, X. Fu, and W. Zhao, “Sto2auc: A stochastic optimal bidding
strategy for microgrids,” IEEE Internet of Things Journal, vol. 4, no. 6, pp. 2260–2274, Dec.
2017.
[25] D. An, Q. Yang, W. Yu, X. Yang, X. Fu, and W.Zhao, “SODA: Strategyproof
online double
auction scheme for multimicrogrids bidding,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 48,
no. 7, pp. 1177–1190, Jul. 2018.
[26] M. N. Faqiry and S. Das, “Double auction with hidden user information: Application to
energy transaction in microgrid,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 49, no. 11, pp.
2326–2339, Nov. 2019.
[27] D. Li, Q. Yang, W. Yu, D. An, Y. Zhang, and W. Zhao, “Towards differential privacybased
online double auction for smart grid,” IEEE Trans. Inf. Forensics Security, vol. 15, pp. 971–
986, Aug. 2019.
[28] M. M. Jalali and A. Kazemi, “Demand side management in a smart grid with multiple electricity
suppliers,” Energy, vol. 81, pp. 766–776, Mar. 2015.
[29] S. Wang, A. F. Taha, J. Wang, K. Kvaternik, and A. Hahn, “Energy crowdsourcing and peertopeer
energy trading in blockchainenabled
smart grids,” IEEE Trans. Syst., Man, Cybern.,
Syst., vol. 49, no. 8, pp. 1612–1623, Aug. 2019.
[30] L. Gkatzikis, I. Koutsopoulos, and T. Salonidis, “The role of aggregators in smart grid demand
response markets,” IEEE J. Sel. Areas Commun., vol. 31, no. 7, pp. 1247–1257, Jun. 2013.
[31] D. F. Spulber, Market Microstructure: Intermediaries and the Theory of the Firm. Cambridge:
Cambridge University Press, 1999.
[32] Q. Lambert, “Business models for an aggregator,” Master’s thesis, KTH Royal Institute of
Technology, 2012.
[33] “Commercial aggrgegation service providers,” National Grid, Tech. Rep., 2015.
[34] e2m. [Online]. Available: https://www.e2m.energy/en/businessmodel.
html
[35] X. Lin, Y. Wang, and M. Pedram, “Designing the optimal pricing policy for aggregators in
the smart grid,” in Proc. IEEE Green Tech. Conf., Corpus Christi, TX, USA, Apr. 2014, pp.
75–80.
[36] Y. Okajima, K. Hirata, T. Murao, T. Hatanaka, V. Gupta, and K. Uchida, “Strategic behavior
and market power of aggregators in energy demand networks,” in Proc. IEEE Conf. Decision
and Control, Melbourne, Australia, Dec. 2017, pp. 694–701.
[37] Z. Li, L. Chen, and G. Nan, “Smallscale
renewable energy source trading: A contract theory
approach,” IEEE Trans. Ind. Informat., vol. 14, no. 4, pp. 1491–1500, Apr. 2018.
[38] D. Li, W.Y.
Chiu, H. Sun, and H. V. Poor, “Multiobjective optimization for demand side
management program in smart grid,” IEEE Trans. Ind. Informat., vol. 14, no. 4, pp. 1482–
1490, Apr. 2018.
[39] E. Du, N. Zhang, B. Hodge, Q. Wang, C. Kang, B. Kroposki, and Q. Xia, “The role of
concentrating solar power toward high renewable energy penetrated power systems,” IEEE
Trans. Power Syst., vol. 33, no. 6, pp. 6630–6641, Nov. 2018.
[40] T. Mai, M. M. Hand, S. F. Baldwin, R. H. Wiser, G. L. Brinkman, P. Denholm, D. J. Arent,
G. Porro, D. Sandor, D. J. Hostick, M. Milligan, E. A. DeMeo, and M. Bazilian, “Renewable
electricity futures for the United States,” IEEE Trans. Sustain. Energy, vol. 5, no. 2, pp. 372–
378, Apr. 2014.
[41] M. Ram, D. Bogdanov, A. Aghahosseini, A. Gulagi, S. Oyewo, M. Child, U. Caldera,
K. Sadovskaia, J. Farfan, L. Barbosa, M. Fasihi, S. Khalili, and C. Breyer, “Global energy
system based on 100% renewable energy –energy transition in europe across power, heat,
transport and desalination sectors,” LUT University and Energy Watch Group, Berlin, Germany,
Tech. Rep., Mar. 2019.
[42] M. T. GarcíaAlvarez
and R. M. MarizPérez,
“Analysis of the success of feedin
tariff for
renewable energy promotion mechanism in the EU: Lessons from Germany and Spain,”
ProcediaSocial
and Behavioral Sciences, vol. 65, pp. 52–57, 2012.
[43] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, Massachusetts,
London, England: MIT Press, 2018.
[44] A. Qazi, F. Hussain, N. A. Rahim, G. Hardaker, D. Alghazzawi, K. Shaban, and K. Haruna,
“Towards sustainable energy: A systematic review of renewable energy sources, technologies,
and public opinions,” IEEE Access, vol. 7, pp. 63 837–63 851, May 2019.
[45] K. D. Orwig, M. L. Ahlstrom, V. Banunarayanan, J. Sharp, J. M. Wilczak, J. Freedman,
S. E. Haupt, J. Cline, O. Bartholomy, H. F. Hamann, B. Hodge, C. Finley, D. Nakafuji, J. L.
Peterson, D. Maggio, and M. Marquis, “Recent trends in variable generation forecasting and
its value to the power system,” IEEE Trans. Sustain. Energy, vol. 6, no. 3, pp. 924–933, Jul.
2015.
[46] J. R. Andrade and R. J. Bessa, “Improving renewable energy forecasting with a grid of numerical
weather predictions,” IEEE Trans. Sustain. Energy, vol. 8, no. 4, pp. 1571–1580,
Oct. 2017.
[47] Y. S. Manjili, R. Vega, and M. M. Jamshidi, “Dataanalyticbased
adaptive solar energy forecasting
framework,” IEEE Syst. Journal, vol. 12, no. 1, pp. 285–296, Mar. 2018.
[48] S. Teleke, M. E. Baran, S. Bhattacharya, and A. Q. Huang, “Rulebased
control of battery
energy storage for dispatching intermittent renewable sources,” IEEE Trans. Sustain. Energy,
vol. 1, no. 3, pp. 117–124, Oct. 2010.
[49] P. Harsha and M. Dahleh, “Optimal management and sizing of energy storage under dynamic
pricing for the efficient integration of renewable energy,” IEEE Trans. Power Syst., vol. 30,
no. 3, pp. 1164–1181, May 2015.
[50] S. Lakshminarayana, Y. Xu, H. V. Poor, and T. Q. S. Quek, “Cooperation of storage operation
in a power network with renewable generation,” IEEE Trans. Smart Grid, vol. 7, no. 4, pp.
2108–2122, Jul. 2016.
[51] M. Chowdhury, M. Rao, Y. Zhao, T. Javidi, and A. Goldsmith, “Benefits of storage control
for wind power producers in power markets,” IEEE Trans. Sustain. Energy, vol. 7, no. 4, pp.
1492–1505, Oct. 2016.
[52] A. J. Conejo, J. M. Morales, and L. Baringo, “Realtime
demand response model,” IEEE
Trans. Smart Grid, vol. 1, no. 3, pp. 236–242, Dec. 2010.
[53] E. Baeyens, E. Y. Bitar, P. P. Khargonekar, and K. Poolla, “Coalitional aggregation of wind
power,” IEEE Trans. Power Syst., vol. 28, no. 4, pp. 3774–3784, Nov. 2013.
[54] Y. Zhao, J. Qin, R. Rajagopal, A. Goldsmith, and H. V. Poor, “Wind aggregation via risky
power markets,” IEEE Trans. Power Syst., vol. 30, no. 3, pp. 1571–1581[55] H. Khazaei and Y. Zhao, “Indirect mechanism design for efficient and stable renewable energy
aggregation,” IEEE Trans. Power Syst., vol. 34, no. 2, pp. 1033–1042, Mar. 2019.
[56] T. Morstyn, A. Teytelboym, and M. D. Mcculloch, “Bilateral contract networks for peertopeer
energy trading,” IEEE Trans. Smart Grid, vol. 10, no. 2, pp. 2026–2035, Mar. 2019.
[57] A. Paudel, K. Chaudhari, C. Long, and H. B. Gooi, “Peertopeer
energy trading in a
prosumerbased
community microgrid: A gametheoretic
model,” IEEE Trans. Ind. Electron.,
vol. 66, no. 8, pp. 6087–6097, Aug. 2019.
[58] A. G. Bakirtzis and A. C. Tellidou, “Agentbased
simulation of power markets under uniform
and payasbid
pricing rules using reinforcement learning,” in Proc. Power Syst. Conf. and
Exposition, Atlanta, GA, USA.
[59] N. K. Yadav, M. Kumar, D. Sharma, A. Bala, and G. Bhargava, “Development of bidding
strategies using genetic algorithm in deregulated electricity market,” in Proc. Int. Conf. Control,
Computing, Commun. and Materials, Allahbad, India, Oct. 2016, pp. 1–5.
[60] N. Azizan Ruhi, K. Dvijotham, N. Chen, and A. Wierman, “Opportunities for price manipulation
by aggregators in electricity markets,” IEEE Trans. Smart Grid, vol. 9, no. 6, pp.
5687–5698, Nov. 2018.
[61] A. Gosavi, “Reinforcement learning: A tutorial survey and recent advances,” INFORMS J.
Computing, vol. 21, no. 2, pp. 178–192, 2009.
[62] R. Bellman, Dynamic Programming. New York, USA: Dover Publications Inc., 2003.
[63] D. E. Kirk, Optimal Control Theory: An Introduction. New York, USA: Dover Publications,
2012.
[64] S. Chen and R. S. Cheng, “Operating reserves provision from residential users through load
aggregators in smart grid: A game theoretic approach,” IEEE Trans. Smart Grid, vol. 10,
no. 2, pp. 1588–1598, Mar. 2019.
[65] SMARD, Electricity Market Platform in Germany. [Online]. Available: https://www.smard.
de/en/5790
[66] L. Gigoni, A. Betti, E. Crisostomi, A. Franco, M. Tucci, F. Bizzarri, and D. Mucci, “Dayahead
hourly forecasting of power generation from photovoltaic plants,” IEEE Trans. Sustain.
Energy, vol. 9, no. 2, pp. 831–842, Apr. 2018.
[67] Y. Tang, J. Ling, T. Ma, N. Chen, X. Liu, and B. Gao, “A game theoretical approach based
bidding strategy optimization for power producers in power markets with renewable electricity,”
Energies, vol. 10, no. 5, p. 627, May 2017.
[68] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic
algorithm: NSGAII,”
IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182–197, Aug. 2002.
[69] Q. Zhang and H. Li, “MOEA/D: A multiobjective evolutionary algorithm based on decomposition,”
IEEE Trans. Evol. Comput., vol. 11, no. 6, pp. 712–731, Nov. 2007.