相較於其他自然語言處理任務，歌詞生成仍鮮少被研究。而當前最尖端的中文語言模型BERT（雙向編碼轉換器）能將成功捕捉語句間詞彙的語意，並且經過大量語料訓練後在克漏字和預測下一句兩個任務中達到令人驚豔的正確率。因此，我們決定借用其能力填詞。我們提出自動判別歌詞分段的演算法以產生歌詞模板，讓BERT大量學習中文歌詞後，在維持原曲架構和詞句長度的情況下以取代字詞和預測上下句的組合策略創作新歌詞。最後，以BLEU分數和問卷結果來評估模型的表現。

Compared to major natural language processing tasks, lyric generation is relatively less investigated. The cutting-edge Chinese language model of the time, BERT, or Bidirectional Encoder Representations from Transformers, can successfully encode semantics of words in sentences and by training with large corpus can predict masked words and next sentences with amazingly accuracy. We decide to customize BERT’s ability to the composition of lyrics. By fine-tuning with a large lyric corpus, we wish to use BERT to compose the lyrics by substitution and prediction schemes. Given a lyric template generated by our segmentation algorithm, we show that the model can convert the lyric into another new lyrics by keeping the same length of words in the original lyrics but change its content. We demonstrate the performance of our model and schemes by using both the BLEU metric and subjective human evaluations.

摘要 -i
Abstract -ii
Acknowledgment -iii
List of Tables -vi
List of Figures -viii
1 Introduction -1
2 Related Work -4
3 Method -6
3.1 Model -7
3.2 Fine-tuning -9
3.2.1 Task #1: Masked Language Model -9
3.2.2 Task #2: Next Sentence Prediction -9
3.3 Automatic Segmentation of Lyrics -11
3.4 Lyric Generation -15
4 Experiments and Results -20
4.1 Data -20
4.2 Effects of Lyric Segmentation -23
4.3 Evaluations of Generated Lyrics -24
5 Conclusion -28
References -29
Appendix A -32
Appendix B -37
Appendix C -42

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
[2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013.
[3] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
[4] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
[5] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018.
[6] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf, 2018.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[8] Eric Malmi, Pyry Takala, Hannu Toivonen, Tapani Raiko, and Aristides Gionis. Dope-learning: A computational approach to rap lyrics generation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 195–204. ACM, 2016.
[9] Peter Potash, Alexey Romanov, and Anna Rumshisky. Ghostwriter: Using an lstm for automatic rap lyric generation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1919–1924, 2015.
[10] Xing Wu, Zhikang Du, Yike Guo, and Hamido Fujita. Hierarchical attention based long short-term memory for chinese lyric generation. Applied Intelligence, 49(1): 44–52, 2019.
[11] Sung-Hwan Son, Hyun-Young Lee, Gyu-Hyeon Nam, and Seung-Shik Kang. Korean song-lyrics generation by deep learning. In Proceedings of the 2019 4th International Conference on Intelligent Information Technology, pages 96–100. ACM, 2019.
[12] Gabriele Barbieri, Franc ̧ois Pachet, Pierre Roy, and Mirko Degli Esposti. Markov constraints for generating lyrics with style. In Ecai, volume 242, pages 115–120, 2012.
[13] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1587–1596. JMLR. org, 2017.
[14] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[16] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[17] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.