在擁有大數據的大型企業中，票務標記自動化在其最佳性能下利用RNN和CNN等機器學習算法而受到歡迎。小型企業實體（包括初創企業和一些中型企業）很難獲得相同的收益，因為它們通常會遭受資源不足的困擾，例如歷史票證數據集不足以及無法使用價格合理的TPU和GPU。過去的作品通常訴諸於傳統的H-RNN和概率模型的變體來解決票證標籤，但是他們對轉移學習和負擔得起的增強技術的關注較少。 Bert和Xlnet中最新的NLP轉移學習技術是最近很少探討的選項。本文提出了三種簡單且負擔得起的方法，其中包括最新的NLP轉移學習和混合數據增強技術，以應對挑戰。 MDAUlmfit，MDAbert和MDAxlnet這三種方法是基於EDA，反向翻譯，ULMFit，Bert和XLnet技術的精心集成而命名的。 MDAULMFit產生的精度約為98％，MDABert為97％，MDAXLnet為98％。本文不僅對模型進行了定量性能分析，而且還對三種方法的實現進行了定性分析，並與H-RNN，基本ulmfit，基本Bert和Xlnet等基線模型進行了結果比較。

The automation of tickets labelling is popular among big enterprises which possess big data, leveraging machine learning algorithms such as RNN and CNNs at their best performances. Small business entities including startups and some medium size enterprises struggle to acquire the same benefit, as they usually suffer from low resources such as insufficient historical tickets data-set and access to affordable TPUs and GPUs. Past works commonly resort to variation of traditional H-RNN and probabilistic models to address ticket labelling, but they pay less attention to Transfer Learning and affordable augmentation techniques. Recent state of the art NLP transfer learning techniques in Bert and Xlnet, are recent options that are less explored. This paper propose three simple and affordable methods which include recent state of the art NLP Transfer learning and mix data augmentation techniques, to address challenges. The three approaches MDAUlmfit, MDAbert and MDAxlnet named base on the careful integration of the techniques EDA, Back-translation, ULMFit, Bert and XLnet. MDAULMFit produce accuracy of around 98\%, MDABert 97\%, MDAXLnet 98\%. This paper not only present the quantitative performance analysis of the models but also provide qualitative analysis of the three methods in terms of implementation and results comparison with baseline models such as H-RNN, basic ulmfit, basic Bert and Xlnet.

Introduction . . . . . . . . . . . . . . . . . . .1
1.1 Background of ticket labelling study . . . . . . . . .1
1.2 The purpose and scope of the study . . . . . . . . . .3

Related Work . . . . . . . . . . . . . . . . . . .5
2.1 Similar work . . . . . . . . . . . . . . . . .5
2.2 Similar work challenges . . . . . . . . . . . . .6
2.3 Proposed solution concepts . . . . . . . . . . . .6
2.3.1 EDA . . . . . . . . . . . . . . . . . . .7
2.3.2 Back Translation . . . . . . . . . . . . . . .8
2.3.3 ULMFit . . . . . . . . . . . . . . . . . . .9
2.3.4 Bert . . . . . . . . . . . . . . . . . . .10
2.3.5 XLNet . . . . . . . . . . . . . . . . . . .11

Overview . . . . . . . . . . . . . . . . . . . 15
3.1 The study proposed framework . . . . . . . . . . . 15
3.1.1Method one: MDAULMFit . . . . . . . . . . . . . 15
3.1.2Method two: MDABERT . . . . . . . . . . . . . . 16
3.1.3Method three: MDAXLNET . . . . . . . . . . . . . 16

Methodology . . . . . . . . . . . . . . . . . . 19
4.1 The Dataset . . . . . . . . . . . . . . . . 19
4.2 Mix Data Augmentation . . . . . . . . . . . . . 20
4.3 Feature Extraction . . . . . . . . . . . . . . 21
4.3.1 Cleaning ticket text . . . . . . . . . . . . . 21
4.3.2 Tokenising the ticket text . . . . . . . . . . . 21
4.3.3 Bert and XLNet input sequence . . . . . . . . . . 22
4.4 Model Fine-tuning . . . . . . . . . . . . . . 24
4.4.1 MDAULMFit: Method one fine-tuning with ULMFit . . . . . 25
4.4.2 MDABERT: Method two fine-tuning with Bert . . . . . . 26
4.4.3 MDAXLNet: Method three fine-tuning with XLNet . . . . . 26
4.5 Baselines methods . . . . . . . . . . . . . . 29

Results and discussion . . . . . . . . . . . . . . 31
5.1 Tickets analysis . . . . . . . . . . . . . . . 31
5.2 Mix Data Augmentation analysis . . . . . . . . . . 33
5.3 Optimal Learning rates . . . . . . . . . . . . . 35
5.4 Loss Analysis . . . . . . . . . . . . . . . . 38
5.5 Validation performance analysis . . . . . . . . . . 42
5.5.1 F1-score performance per class analysis . . . . . . . 45
5.5.2 Ticket labelings limitation . . . . . . . . . . . 47

Conclusion and future work . . . . . . . . . . . . . 49

Appendix . . . . . . . . . . . . . . . . . . . 54
A Data-sets samples and source code information . . . . . 54

[1] Feras Al-Hawari and Hala Barham. “A machine learning based help desk system for IT service management”. In:Journal of King Saud University-Computer and Information Sciences(2019).

[2] Gilmar Barreto, Paulo D Battaglin, and Sergio Varga. “Ensuring Efficient IT Service Management to Increase Information Systems Availability”. In:Journal of Information Systems Engineering & Management 4.4 (2019), em0108.

[3] G Chin, Y Benslimane, and Z Yang. “Examining the application of standards for information technology service management practice: An empirical study”. In:2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM). IEEE. 2017, pp. 841–845.

[4] Gopinath Krishnan and Vinod Ravindran. “IT service management automation and its impact to IT industry”. In:2017 International Conference on Computational Intelligence in Data Science (ICCIDS). IEEE. 2017, pp. 1–4.

[5] Volodymyr Lyubinets, Taras Boiko, and Deon Nicholas. “Automated labeling of bugs and tickets using attention-based mechanisms in recurrent neural networks”.In:2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP). IEEE. 2018, pp. 271–275.47

[6] Ea-Ee Jan, Kuan-Yu Chen, and Tsuyoshi Id ́e. “Probabilistic text analytics frame-work for information technology service desk tickets”. In:2015 IFIP/IEEE International Symposium on Integrated Network Management (IM). IEEE. 2015,pp. 870–873.

[7] Jason W Wei and Kai Zou. “Eda: Easy data augmentation techniques for boosting performance on text classification tasks”. In:arXiv preprint arXiv:1901.11196(2019).

[8] Rico Sennrich, Barry Haddow, and Alexandra Birch. “Edinburgh neural machine translation systems for wmt 16”. In:arXiv preprint arXiv:1606.02891(2016).

[9] Sergey Edunov et al. “Understanding back-translation at scale”. In:arXiv preprint arXiv:1808.09381(2018).

[10] Zihang Dai et al. “Transformer-xl: Attentive language models beyond a fixed-length context”. In:arXiv preprint arXiv:1901.02860(2019).

[11] Jeremy Howard and Sebastian Ruder. “Universal language model fine-tuning for text classification”. In:arXiv preprint arXiv:1801.06146(2018).

[12] Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers for language understanding”. In:arXiv preprint arXiv:1810.04805(2018).

[13] Ashish Vaswani et al. “Attention is all you need”. In:Advances in neural information processing systems. 2017, pp. 5998–6008.

[14] M Gabri ́e, EW Tramel, and F Krzakala. “Advances in Neural Information Processing Systems”. In: (2015).

[15] Matthew E Peters et al. “Deep contextualized word representations”. In:arXiv preprint arXiv:1802.05365(2018).

[16] Zhilin Yang et al. “Xlnet: Generalized autoregressive pretraining for language understanding”. In:Advances in neural information processing systems. 2019,pp. 5754–5764.

[17] Alec Radford et al. “Improving language understanding by generative pre-training”.In:URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/languageunderstanding paper. pdf(2018).

[18] Xu LIANG.What is XLNet and why it outperforms BERT.URL:https://towardsdatascience . com / what - is - xlnet - and - why - it -outperforms-bert-8d8fce710335. (accessed: 03.04.2020).

[19] ESSEN.AI.What is XLNet and why it outperforms BERT.URL:https://blog.essen.ai/what-is-xlnet-and-how-does-it-work/.(accessed: 03.04.2020).

[20] Chris McCormick.BERT Fine-Tuning Tutorial with PyTorch.URL:http://mccormickml.com/2019/07/22/BERT-fine-tuning/. (accessed:09.09.2019).

[21] Chris McCormick.XLNet Fine-Tuning Tutorial with PyTorch.URL:http://mccormickml.com/2019/09/19/XLNet-fine-tuning/. (accessed:01.02.2020).

[22] Sam Shleifer. “Low resource text classification with ulmfit and back translation”.In:arXiv preprint arXiv:1903.09244(2019).

[23] Cyril Goutte and Eric Gaussier. “A Probabilistic Interpretation of Precision, Re-call and F-Score, with Implication for Evaluation”. In: vol. 3408. Apr. 2005,pp. 345–359.DOI:10.1007/978-3-540-31865-1_25.

[24] Yizhe Zhang et al. “Adversarial feature matching for text generation”. In:Proceedings of the 34th International Conference on Machine Learning-Volume 70.JMLR. org. 2017, pp. 4006–4015.

[25] Xing Wu et al. “Conditional BERT contextual augmentation”. In:International Conference on Computational Science. Springer. 2019, pp. 84–95.