本論文提出一個介系詞改錯方法，可以在不依賴人工標注資料的情況下改正句子中潛在的介系詞錯誤。在我們的方法中，我們在可能遺漏介係詞的位置插入佔位符，並嘗試使用遮罩語言模型來替換或刪除句子中的介系詞和占位符來改正潛在的介系詞錯誤。我們的方法是將母語語料庫中的句子轉換為帶有遮罩的句子和被遮住的介係詞（或符號“[NONE]”）組成的訓練資料，用來表示遺漏、錯誤和多餘的介係詞錯誤，並用合成的資料來訓練遮罩語言模型，使之有能力改正介係詞錯誤。
那些訓練資料是透過遮蓋現有介係詞或在容易出現多餘介係詞的位置插入表示遮蓋的符號來建立的，此外，我們使用 BEA-2019 和 CONLL-2014 的資料集進行評估。初步結果顯示，我們的方法跟前人的研究成果比起來有較好表現。

We introduce a method AccuPrep for correcting preposition errors in a given sentence without using annotated training data. In our approach, we insert placeholders for potential missing prepositions and then attempt to replace or delete prepositions and placeholders with a masked language model (MLM). The method involves converting sentences in a given reference corpus into a dataset of pairs of masked sentence and filler prepositions (or the “[NONE]” symbol) to represent missing, wrong, and unnecessary preposition errors, training a MLM for correcting preposition errors. These masks are created either by replacing existing prepositions or by inserting in potential positions of unnecessary prepositions. We present a prototype based on the proposed method and test on the BEA-2019 shared task and the CONLL-2014 shared task. Preliminary evaluation shows that our approach outperforms previous work.

Abstract i
摘要 ii
致謝 iii
Contents iv
List of Figures vi
List of Tables vii
1 Introduction 1
2 Related Work 4
3 The AccuPrep system 6
3.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Learning to correct preposition errors . . . . . . . . . . . . . . . . . 7
3.2.1 Synthesizing data for missing errors . . . . . . . . . . . . . . 7
3.2.2 Training a Preposition Missing Error Detection Model . . . 8
3.2.3 Synthesizing data for unnecessary errors . . . . . . . . . . . 9
3.2.4 Synthesizing data for replacement errors . . . . . . . . . . . 10
3.2.5 Finetuning a Masked Language Model . . . . . . . . . . . . 11
3.3 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4 Experiment and Evaluation 14
4.1 Datasets and Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2 Evaluation Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.3 Experiment Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.4 Models compared . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5 Results and Discussion 18
5.1 Results from the CONLL-2014 shared task dataset evaluation . . . 18
5.2 Result from the BEA-2019 shared task evaluation . . . . . . . . . . 19
5.3 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6 Conclusion and Future Work 23
Reference 24

[1] Christopher Bryant and Ted Briscoe. Language Model Based Grammatical Error Correction without Annotated Training Data. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 247–253, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-0529. URL https: //aclanthology.org/W18-0529.

[2] Christopher Bryant, Mariano Felice, and Ted Briscoe. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1074. URL https://www.aclweb.org/anthology/P17-1074.

[3] Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. The BEA-2019 Shared Task on Grammatical Error Correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4406. URL https: //aclanthology.org/W19-4406.

[4] Yo Joong Choe, Jiyeon Ham, Kyubyong Park, and Yeoil Yoon. A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 213–227, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4423. URL https://aclanthology.org/W19-4423.

[5] Daniel Dahlmeier and Hwee Tou Ng. Better Evaluation for Grammatical Error Correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572, Montr´eal, Canada, June 2012. Association for Computational Linguistics. URL https://aclanthology.org/N12-1067

[6] Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 15–24, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-1702. URL https://aclanthology.org/W14-1702.

[7] Kenneth Heafield. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187– 197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123.

[8] Ruobing Li, Chuan Wang, Yefei Zha, Yonghong Yu, Shiman Guo, Qiang Wang, Yang Liu, and Hui Lin. The LAIX Systems in the BEA-2019 GEC Shared Task. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 159–167, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4416. URL https://aclanthology.org/W19-4416.

[9] Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. Mining Revision Log of Language Learning SNS for Automated Japanese Error Correction of Second Language Learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–155, Chiang Mai, Thailand, November 2011. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I11-1017.

[10] Daniel Naber. A Rule-Based Style and Grammar Checker. page 76, August 2003.

[11] Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. The CoNLL-2014 Shared Task on Gram matical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-1701. URL https://www.aclweb.org/anthology/W14-1701.

[12] Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem N. Chernodub, and Oleksandr Skurzhanskyi. GECToR - Grammatical Error Correction: Tag, Not Rewrite. In BEA@ACL, 2020. doi: 10.18653/v1/2020.bea-1.16.

[13] Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. The Illinois-Columbia System in the CoNLL-2014 Shared Task. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 34–42, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-1704. URL https: //aclanthology.org/W14-1704.

[14] Chuan Wang, Ruobing Li, and Hui Lin. Deep Context Model for Grammatical Error Correction. In 7th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2017), pages 167–171. ISCA, August 2017. doi: 10.21437/ SLaTE.2017-29. URL https://www.isca-speech.org/archive/slate_2017/ wang17_slate.html.

[15] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6.

[16] Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, and Andrew Y. Ng. Neural Language Correction with Character-Based Attention. ArXiv, 2016.

[17] Zheng Yuan and Ted Briscoe. Grammatical error correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380–386, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-1042. URL https://aclanthology.org/N16-1042.

[18] Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. arXiv:1506.06724 [cs], June 2015. URL http://arxiv.org/abs/1506.06724. arXiv: 1506.06724.