隨著資料越來越大，資料庫系統需要處理的資料也越來越多，近年常被使用的解決方式就是利用分散式資料庫系統來將資料存放於多台電腦上，正確的資料切割（data partition）就會變得相當重要。而如今資料庫的工作負載（workload）與過去傳統的情境不同，時常會有一些動態的變化，若我們使用固定的data partition，可能導致機器的負載（load）隨著流量變化而分布不均，因此當 workload 變化時，我們就需要為資料做 re-partitioning來平均各個機器的負載（load balancing）。
因爲 workload 太複雜，又需要同時考慮到 data partition、load balance以及最大化資料庫系統的 throughput，因此希望能藉由機器學習的方法幫助我們解決這個問題。我們使用機器學習中的增強式學習（Reinforcement Learning，RL）來解決routing 問題，藉由做出routing 決定來做到 data re-partitioning 以及 load balancing，並使用 RL 從過去的 routing 中獲得回饋，來改進對未來 transaction 的 routing 決策。我們使用四種RL 的作法，分別為 Online RL、Offline RL、Bootstrap RL 以及情境式賭博機算法（Contextual Bandits）。
本篇論文分析了這四種 RL 方式在理論上的關鍵差異，並在實驗之中使用了標準的 TPC-C 測試模組搭配複雜的動態 workload 進行測試，證實 RL 方式解決了過去作法中的漏洞，讓 throughput 得到了顯著的提升，同時也找出了各種 RL 方式各自適用的情境。

As the data becomes larger and larger, the database system needs to process more and more data today. The solution often used in recent years is to use a distributed database system to store data on multiple machines. Correctly partitioning the data becomes quite important. However, the workload today is different from the traditional situation in the past. There are often some dynamic changes in workload patterns. If we use a fixed data partition, the machine's load may be unbalanced. Therefore, when the workload changes, We need to do re-partitioning for the data to balance the load of each machine.
The workload is too complex, and it is necessary to consider data partition, balance the load, and maximize the throughput of the database system at the same time. In this case, we hope that machine learning can help us solve this problem. We use reinforcement learning to help us solve the routing problem, by making routing decisions to achieve data re-partitioning and load balancing, and get feedback from past routing to improve routing decisions for future transactions. We use four RL methods, online RL, offline RL, bootstrap RL, and contextual bandits.
This paper analyzes the key differences in theory between these four RL methods. We use the standard TPC-C benchmark with the complex dynamic workload for testing in the experiment, and confirm that the RL method solves the drawbacks in the past solution, so that the throughput has been significantly improved. At the same time, we also find out the applicable scenarios of various RL methods.

摘要 2
ABSTRACT 3
致謝 4
目錄 5
第1章 INTRODUCTION 6
第2章 BACKGROUND 9
第1節 DETERMINISTIC DATABASE SYSTEMS 9
第2節 HERMES 10
第3章 MAIN IDEA 12
第1節 PROBLEM FORMULATION 12
第2節 MARKOV DECISION PROCESS 12
第3節 ONLINE REINFORCEMENT LEARNING 14
第4節 OFFLINE REINFORCEMENT LEARNING 16
第5節 BOOTSTRAP REINFORCEMENT LEARNING 18
第6節 CONTEXTUAL BANDITS 19
第7節 比較表 20
第4章 PRACTICAL OPTIMIZATION 21
第5章 EXPERIMENTS 22
第1節 實驗設定 22
第2節 各個方法在複雜的 WORKLOAD 下的表現 23
第3節 各個 RL 方法之間的比較 26
第4節 SENSITIVITY 29
第6章 RELATED WORKS 32
第7章 CONCLUSION 33
第8章 REFERENCES 33

1] Ren, K., Thomson, A., & Abadi, D. J. (2014). An evaluation of the advantages and disadvantages of deterministic database systems. Proceedings of the VLDB Endowment, 7(10), 821-832.
[2] Thomson, A., & Abadi, D. J. (2010). The case for determinism in database systems. Proceedings of the VLDB Endowment, 3(1-2), 70-80.
[3] Lin, Y. S., Pi, S. K., Liao, M. K., Tsai, C., Elmore, A., & Wu, S. H. (2019). MgCrab: transaction crabbing for live migration in deterministic database systems. Proceedings of the VLDB Endowment, 12(5), 597-610.
[4] Lin, Y. S., Tsai, C., Lin, T. Y., Chang, Y. S., & Wu, S. H. (2021, June). Don't Look Back, Look into the Future: Prescient Data Partitioning and Migration for Deterministic Database Systems. In Proceedings of the 2021 International Conference on Management of Data (pp. 1156-1168).
[5] Serafini, M., Taft, R., Elmore, A. J., Pavlo, A., Aboulnaga, A., & Stonebraker, M. (2016). Clay: Fine-grained adaptive partitioning for general database schemas. Proceedings of the VLDB Endowment, 10(4), 445-456.
[6] Taft, R., Mansour, E., Serafini, M., Duggan, J., Elmore, A. J., Aboulnaga, A., ... & Stonebraker, M. (2014). E-store: Fine-grained elastic partitioning for distributed transaction processing systems. Proceedings of the VLDB Endowment, 8(3), 245-256.
[7] Thomson, A., Diamond, T., Weng, S. C., Ren, K., Shao, P., & Abadi, D. J. (2012, May). Calvin: fast distributed transactions for partitioned database systems. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (pp. 1-12).
[8] Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., ... & Tatbul23, N. Neo: A Learned Query Optimizer. Proceedings of the VLDB Endowment, 12(11).
[9] Marcus, R., Negi, P., Mao, H., Tatbul, N., Alizadeh, M., & Kraska, T. (2022). Bao: Making learned query optimization practical. ACM SIGMOD Record, 51(1), 6-13.
[10] Yang, Z., Chiang, W. L., Luan, S., Mittal, G., Luo, M., & Stoica, I. (2022). Balsa: Learning a Query Optimizer Without Expert Demonstrations. arXiv preprint arXiv:2201.01441.
[11] Ortiz, J., Balazinska, M., Gehrke, J., & Keerthi, S. S. (2018, June). Learning state representations for query optimization with deep reinforcement learning. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning (pp. 1-4).
[12] Trummer, I., Moseley, S., Maram, D., Jo, S., & Antonakakis, J. (2018). Skinnerdb: regret-bounded query evaluation via reinforcement learning. Proceedings of the VLDB Endowment, 11(12), 2074-2077.
[13] Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., & Stoica, I. (2018). Learning to optimize join queries with deep reinforcement learning. arXiv preprint arXiv:1808.03196.
[14] Marcus, R., & Papaemmanouil, O. (2018, June). Deep reinforcement learning for join order enumeration. In Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (pp. 1-4).
[15] Altman, E. (1999). Constrained Markov decision processes: stochastic modeling. Routledge.
[16] Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
[17] Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine learning, 8(3), 279-292.
[18] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
[19] Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
[20] Lange, S., Gabel, T., & Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement learning (pp. 45-73). Springer, Berlin, Heidelberg.
[21] Fujimoto, S., Conti, E., Ghavamzadeh, M., & Pineau, J. (2019). Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708.
[22] Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov), 397-422.
[23] Chapelle, O., & Li, L. (2011). An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24.
[24] Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010, April). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web (pp. 661-670).
[25] Wang, J., Trummer, I., & Basu, D. (2021). UDO: universal database optimization using reinforcement learning. arXiv preprint arXiv:2104.01744.
[26] Sharma, A., Schuhknecht, F. M., & Dittrich, J. (2018). The case for automatic database administration using deep reinforcement learning. arXiv preprint arXiv:1801.05643.
[27] Zhang, J., Liu, Y., Zhou, K., Li, G., Xiao, Z., Cheng, B., ... & Li, Z. (2019, June). An end-to-end automatic cloud database tuning system using deep reinforcement learning. In Proceedings of the 2019 International Conference on Management of Data (pp. 415-432).
[28] Li, G., Zhou, X., Li, S., & Gao, B. (2019). Qtune: A query-aware database tuning system with deep reinforcement learning. Proceedings of the VLDB Endowment, 12(12), 2118-2130.
[29] Gur, Y., Yang, D., Stalschus, F., & Reinwald, B. (2021, March). Adaptive Multi-Model Reinforcement Learning for Online Database Tuning. In EDBT (pp. 439-444).
[30] Cereda, S., Valladares, S., Cremonesi, P., & Doni, S. (2021). CGPTuner: a contextual gaussian process bandit approach for the automatic tuning of IT configurations under varying workload conditions. Proceedings of the VLDB Endowment, 14(8), 1401-1413.
[31] Cai, B., Liu, Y., Zhang, C., Zhang, G., Zhou, K., Liu, L., ... & Xing, J. (2022, June). HUNTER: An Online Cloud Database Hybrid Tuning System for Personalized Requirements. In Proceedings of the 2022 International Conference on Management of Data (pp. 646-659).
[32] Sadri, Z., Gruenwald, L., & Leal, E. (2020, April). Online index selection using deep reinforcement learning for a cluster database. In 2020 IEEE 36th International Conference on Data Engineering Workshops (ICDEW) (pp. 158-161). IEEE.
[33] Sadri, Z., Gruenwald, L., & Lead, E. (2020, August). DRLindex: deep reinforcement learning index advisor for a cluster database. In Proceedings of the 24th Symposium on International Database Engineering & Applications (pp. 1-8).
[34] Lan, H., Bao, Z., & Peng, Y. (2020, October). An index advisor using deep reinforcement learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 2105-2108).
[35] Licks, G. P., & Meneguzzi, F. (2020). Automated database indexing using model-free reinforcement learning. arXiv preprint arXiv:2007.14244.
[36] Kossmann, J., Kastius, A., & Schlosser, R. (2022). SWIRL: Selection of Workload-aware Indexes using Reinforcement Learning. In EDBT (pp. 2-155).
[37] Wu, W., Wang, C., Siddiqui, T., Wang, J., Narasayya, V., Chaudhuri, S., & Bernstein, P. A. (2022, June). Budget-aware Index Tuning with Reinforcement Learning. In Proceedings of the 2022 International Conference on Management of Data (pp. 1528-1541).
[38] Marcus, R., & Papaemmanouil, O. (2018, June). Deep reinforcement learning for join order enumeration. In Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (pp. 1-4).
[39] Yu, X., Li, G., Chai, C., & Tang, N. (2020, April). Reinforcement learning with tree-lstm for join order selection. In 2020 IEEE 36th International Conference on Data Engineering (ICDE) (pp. 1297-1308). IEEE.
[40] Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., & Stoica, I. (2018). Learning to optimize join queries with deep reinforcement learning. arXiv preprint arXiv:1808.03196.
[41] Sioulas, P., & Ailamaki, A. (2021, June). Scalable Multi-Query Execution using Reinforcement Learning. In Proceedings of the 2021 International Conference on Management of Data (pp. 1651-1663).
[42] Hilprecht, B., Binnig, C., & Röhm, U. (2020, June). Learning a partitioning advisor for cloud databases. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (pp. 143-157).