本研究旨在透過強化式學習並整合重心估測與力矩控制提昇雙足機器人行走穩定性。本實驗室原已利用重心自適應控制器完成雙足機器人之平衡控制，其利用滑車模型(Cart-Table Model)作為機器人動態模型，並以串聯彈性致動器(Series Elastic Actuator)構成的踝關節作為力矩輸入，再透過重心自適應控制器將實際系統存在的重心偏差納入考量，完成平衡控制。本研究以此為基礎，將擺動腳質量加入考慮，設計雙滑車模型用於機器人行走模型之規劃，將機器人質心軌跡及擺動腳軌跡同時規劃，使機器人能以貼近自身動態的軌跡向前行走。此外，由於機器人行走時腳部擺動造成的角動量使機器人行走方向不穩定，本研究亦提出一增強式學習架構，利用手部擺動以及髖關節旋轉使機器人行走方向更加穩定。

This thesis aims to improve walking stability in bipedal robots through integrate torque control and CoG estimation. In addition, we utilize reinforcement learning to make walking direction of the robot more straight. The cart table model is used as dynamic model of the robot. In order to use ankle torque as torque input of the system, the ankle of the robot is consisting of series elastic actuator. Because of the change of robot’s posture or measurement errors, CoG may not be at the ideal place. So, we use a controller that can automatically estimate the bias of the CoG. Based on the dynamic model and the controller above. We take the mass of the swing leg into consideration and propose a model named "double cart-table model" to generate walking pattern. The double cart table model simultaneously plans the robot’s upper body centroid and the end point of swing leg trajectory by using two equivalent masses. In this way, the robot can walk forward with a trajectory which is close to its own dynamics. Furthermore, the angular momentum caused by swing leg during walking period makes the walking direction of the robot unstable. This study then proposes a reinforcement learning method to make it more stable.

摘要 I
Abstract II
目錄 III
圖目錄 VI
表目錄 X
符號表 XI
第一章 緒論 1
1.1 研究動機與目的 1
1.2 文獻回顧 4
1.3 論文簡介 9
第二章 系統動態與控制 10
2.1 系統動態模型 10
2.2 控制器設計 14
2.3 模擬結果 16
第三章 雙足機器人行走模型 19
3.1 行走模型推導 19
3.2 考量擺動腳之行走模型推導 20
3.3 模擬結果 28
3.4 雙腳支撐周期 33
3.5 機器人質心與擺動腳質心偏差 36
第四章 增強式學習應用於機器人行走 38
4.1 增強式學習 38
4.2 時序差分學習 40
4.3 系統架構 41
第五章 機器人實作 43
5.1 硬體架構 43
5.2 串聯彈性致動器 46
5.3 平衡控制器驗證 51
5.4 行走模型驗證 54
5.5 增強式學習 61
第六章 結論與未來工作 64
6.1 結論 64
6.2 未來工作 64
參考文獻 66

[1] "Pepper," Aldebaran Robotics, SoftBank Robotics Corp., [Online]. Available:https://www.ald.softbankrobotics.com/en/coolrobots-/pepper.
[2] "Zenbo," ASUSTeK Computer Inc., [Online]. Available: https://zenbo.asus.com/tw/.
[3] T. McGeer, "Passive dynamic walking," The international journal of robotics research, vol. 2, no. 9, pp. 62-82, 1990.
[4] "ASIMO," Honda Motor Co., Ltd., [Online]. Available: http://asimo.honda.com/.
[5] "Atlas," Boston Dynamics, [Online]. Available: https://www.bostondynamics.com/atlas.
[6] Kajita, Shuuji and Hirukawa, Hirohisa and Harada, Kensuke and Yokoi, Kazuhito, Introduction to humanoid robotics, Springer, 2014.
[7] C. F. Huang, T. J. Yeh, "Control of a Pedaled, Self-balanced Unicycle with Adaptation Capability," in International Conference on Informatics in Control, Automation and Robotics (ICINCO), Vienna, 2014.
[8] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, “Playing Atari with Deep Reinforcement Learning,” DeepMind Technologies, 2013.
[9] P. A. Gill and M. M. Williamson, "Series elastic actuators," in Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems 95.'Human Robot Interaction and Cooperative Robots'.
[10] M. M. Williamson, Series elastic actuators, 1995.
[11] B. Ugurlu, N. G. Tsagarakis, E. Spyrakos-Papastavridis and D. G. Caldwell, "Compliant joint modification and real-time dynamic walking implementation on bipedal robot cCub," in 2011 IEEE International Conference on Mechatronics, Istanbul, 2011.
[12] X. Huo, Y. Xia, Y. Liu, L. Jiang and H. Liu, "Humanoid arm with the integrated serial elastic actuator," in 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, 2014.
[13] "HEBI ROBOTICS Product," HEBI ROBOTICS, [Online]. Available: http://hebirobotics.com/products/.
[14] Vukobratovic, Miomir and Borovac, Branislav and Surla, Dusan and Stokic, Dragan, Biped locomotion: dynamics, stability, control and application, Springer Science & Business Media, 2012.
[15] Hirai, Kazuo and Hirose, Masato and Haikawa, Yuji and Takenaka, Toru, "The development of Honda humanoid robot," in IEEE International Conference on Robotics and Automation, 1998.
[16] Bhounsule, Pranav A and Cortell, Jason and Grewal, Anoop and Hendriksen, Bram and Karssen, JG Daniel and Paul, Chandana and Ruina, Andy, "Low-bandwidth reflex-based control for lower power walking: 65 km on a single battery charge," The International Journal of Robotics Research, vol. 10, no. 33, pp. 1305-1321, 2014.
[17] Westervelt, Eric R and Grizzle, Jessy W and Chevallereau, Christine and Choi, Jun Ho and Morris, Benjamin, Feedback control of dynamic bipedal robot locomotion, CRC press, 2007.
[18] Grizzle, Jessy W and Chevallereau, Christine and Ames, Aaron D and Sinnet, Ryan W, "3D bipedal robotic walking: models, feedback control, and open problems," IFAC Proceedings Volumes, vol. 14, no. 43, pp. 505-532, 2010.
[19] KAO-SHING HWANG, JIN-LING LIN and JHE-SYUN LI, “Biped Balance Control by Reinforcement Learning,” JOURNAL OF INFORMATION SCIENCE AND ENGINEERING , pp. 1041-1060, 2016.
[20] Jun Morimoto, Gordon Cheng, “A Simple Reinforcement Learning Algorithm For Biped Walking,” 於 2004 IEEE International Conference on Robotics & Automation, New Orleans, 2004.
[21] P. Wawrzynski, "Autonomous Reinforcement Learning with Experience Replay for Humanoid Gait Optimization," Procedia Computer Science, pp. 205-211, 2012.
[22] T. Otani, K. Hashimoto, S. Miyamae, H. Ueta, M.Sakaguchi, Y. Kawakami, H.O. Lim, "Angular Momentum Compensation in Yaw Direction using Upper Body based on Human Running," in 2017 IEEE International Conference on Robotics and Automation, Singapore, 2017.
[23] C. F. Huang, Center of Gravity Adaptation and anti-slip Control for a Pedaled Self-balanced Unicycle, 2016.
[24] Xiuping Mu and Qiong Wu, "Sagittal Gait Synthesis for a Five-Link Biped Robot," in 2004 American Control Conference, Boston, 2004.