通訊科技快速發展，為因應網際網路的興起與智慧行動裝置的高普及率，企業的行銷方式隨之改變。發送廣告的媒介從原先的簡訊、電子郵件逐漸地被取代為網頁推播。推播通知的主動性和即時性大大的增加廣告的曝光率，進而提高用戶對企業商品的觸及率，它在電子商務中是一種經濟效益高的快速營銷策略。但網頁推播並不是一味的發送廣告即可獲得良好的行銷效果，用戶在推播中處於被動狀態，用戶在何時接收何種內容全由企業決定，在這種企業主導的情況下所發送的商品訊息未必與用戶的需求相符。訊息發送的時間點對於用戶來說如果不合宜，將會導致用戶忽略該則推播通知。尤有甚者，頻繁地推送可能使用戶感到厭煩、壓迫，甚至可能造成客戶的流失並對企業造成形象或利益上的損害。
本論文中，透過與企業「禾多移動多媒體股份有限公司(Likr)」合作，提出基於深度學習的方法的「封鎖意圖預測系統」。系統架構包含三個部分：特徵轉換層，歷史行為萃取層和判別層，特徵轉換層中利用文字探勘與文本分類的技術分析推播通知的內文；歷史行為萃取層中透過遞迴神經網路來分析用戶近期接收推播通知的行為紀錄，並結合注意力機制獲取每則推播通知對用戶的影響程度，最後通過判別層來預測下一則推播通知的內容與推播的時機是否會造成用戶對於推播服務進行封鎖。「封鎖意圖預測系統」的實驗結果中顯示，系統更新後未來的三日內預測 (AUROC) 最高可以達0.7976。實驗結果驗證，即使處於高度變化的時序列推播資料中，該模型確實可以精準地預測封鎖率。本論文期望透過Likr公司的合作計畫，運用本論文所提出之「封鎖意圖預測系統」進行自適的調度推播通知。在用戶注意力有限的情況下，有效地減少封鎖率，提升用戶對產品的參與度並帶來更高的點擊率，進而提高企業形象與客戶滿意度。

With the rapid development of communication technology, the rise of the Internet, and mobile devices' popularity, companies' marketing strategies have changed accordingly. The way for sending advertisements has gradually been replaced by web push, instead of SMS and e-mail. The initiative and immediacy of push notifications have significantly increased advertisements' exposure and the user's reach of certain products. It is an economical and efficient marketing strategy in e-commerce. However, it is impossible to get satisfying results all the time by sending notifications at will. In the process of Web push, users are passive. When and what kind of content users receive are decided by companies. In this case, the product messages sent by companies may not meet the needs of users. If the pushed message is inappropriate for the users, they will ignore the push notification. Even worse, frequent pushes may also make users feel annoyed and oppressed, resulting in customer churn and damage to the company's image.
In this thesis, we cooperated with the largest push notification company in Asia, Likr, and proposed a “blockade intention prediction system” based on deep learning methods. The system consists of three parts: feature transformation layer, historical behavior extraction layer, and discrimination layer. In the feature transformation layer, text mining techniques are used to analyze the push notification content. In the historical behavior extraction layer, we used recurrent neural networks to investigate user’s recent behavior records receiving push notifications and the attention mechanism to capture each push notification's impacts on users. Finally, we use the discriminant layer to predict user blockade. The result shows that the prediction can be as high as 0.7976 (AUROC), even in highly variable time series push data. It has been shown that the model can indeed increase user satisfaction for products and reduce the block rate. Through our findings, Likr can schedule push notifications adaptively based on the predicted user blockage intention. The user’s attention is limited. We hope our system can reduce user blockage, improve users' interest in the product, and increase customer satisfaction.

誌謝 i
中文摘要 ii
Abstract iv
目錄 vi
圖目錄 viii
表目錄 ix
第一章研究背景與動機 1
1.1 研究背景 1
1.2 動機 3
1.3 研究貢獻 5
1.4論文架構 6
第二章資料集介紹 7
2.1 特徵說明 8
2.2 有效的推播時間 10
第三章研究方法 13
3.1 深度神經網路Deep Neural Network 13
3.2 門控制循環單元Gate Recurrent Unit 15
3.3 序列到序列模型 Sequence-to-sequence 16
3.4注意力機制 Attention Mechanism 18
3.5 正則化 Regularization 19
3.5.1 Weight decay 19
3.5.2 Dropout 19
3.5.3 Batch Normalization 21
3.5.4 Layer Normalization 22
3.6自然語言處理 23
3.6.1 Latent Dirichlet Allocation 23
3.6.2 詞向量 24
第四章研究模型架構 28
4.1概述 28
4.2 推播內文轉換 29
4.2.1 蒐集語料 29
4.2.2 資料前處理 29
4.2.3 內文嵌入向量模型 30
4.3 時間特徵轉換 32
4.3.1 時間類別特徵轉換 32
4.3.2 時間數值特徵轉換 33
4.4歷史行為萃取層 35
4.5封鎖行為判別層 36
第五章結果與討論 38
5.1實驗設置 38
5.2評估準則 39
5.2.1 Receiver Operating Characteristics 39
5.2.2 Precision-Recall Curve 39
5.2.3 F1-score 40
5.3 模型細節 40
5.3.1 分析用戶行為序列窗 40
5.3.2 分析樣本行為序列窗 41
5.3.3 小節 42
5.4文字嵌入模型比較 42
5.5分析封鎖的因素 44
5.6分析模型對未來幾天的預測結果 45
5.7不同的資料劃分比較 47
第六章結論與未來發展 49
參考資料 51

[1] K. M. Dubas and C. Sylla, “Webcasting/Push Technology on Intranets and Extranets,” Prod. Oper. Manag. Soc., pp. 112–120, Mar. 2001.
[2] K. M. Bell, D. N. Bleau, and J. T. Davey, “Push notification service,” US8064896B2, Nov. 22, 2011.
[3] J. L. Herlocker, J. A. Konstan, A. Borchers, and J. Riedl, “An algorithmic framework for performing collaborative filtering,” in Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA, Aug. 1999, pp. 230–237, doi: 10.1145/312624.312682.
[4] H.-T. Cheng et al., “Wide & Deep Learning for Recommender Systems,” in Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, New York, NY, USA, Sep. 2016, pp. 7–10, doi: 10.1145/2988450.2988454.
[5] H. Guo, R. Tang, Y. Ye, Z. Li, and X. He, “DeepFM: A Factorization-Machine based Neural Network for CTR Prediction,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia, Aug. 2017, pp. 1725–1731, doi: 10.24963/ijcai.2017/239.
[6] G. de Souza Pereira Moreira, F. Ferreira, and A. M. da Cunha, “News Session-Based Recommendations using Deep Neural Networks,” in Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems, New York, NY, USA, Oct. 2018, pp. 15–23, doi: 10.1145/3270323.3270328.
[7] S. Okura, Y. Tagami, S. Ono, and A. Tajima, “Embedding-based News Recommendation for Millions of Users,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, Aug. 2017, pp. 1933–1942, doi: 10.1145/3097983.3098108.
[8] G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006, doi: 10.1126/science.1127647.
[9] J. Heaton, “Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep learning,” Genet. Program. Evolvable Mach., vol. 19, no. 1, pp. 305–307, Jun. 2018, doi: 10.1007/s10710-017-9314-z.
[10] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling,” ArXiv14123555 Cs, Dec. 2014, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1412.3555.
[11] K. Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, Oct. 2014, pp. 1724–1734, doi: 10.3115/v1/D14-1179.
[12] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 3104–3112.
[13] S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, “Generating Sentences from a Continuous Space,” in Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany, Aug. 2016, pp. 10–21, doi: 10.18653/v1/K16-1002.
[14] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ArXiv14090473 Cs Stat, May 2016, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1409.0473.
[15] V. Mnih, N. Heess, A. Graves, and koray kavukcuoglu, “Recurrent Models of Visual Attention,” in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 2204–2212.
[16] J. Ba, V. Mnih, and K. Kavukcuoglu, “Multiple Object Recognition with Visual Attention,” ArXiv14127755 Cs, Apr. 2015, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1412.7755.
[17] K. Gregor, I. Danihelka, A. Graves, D. Rezende, and D. Wierstra, “DRAW: A Recurrent Neural Network For Image Generation,” in International Conference on Machine Learning, Jun. 2015, pp. 1462–1471, Accessed: Sep. 21, 2020. [Online]. Available: http://proceedings.mlr.press/v37/gregor15.html.
[18] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, Jan. 2014.
[19] S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, Lille, France, Jul. 2015, pp. 448–456, Accessed: Sep. 21, 2020. [Online].
[20] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” presented at the Advances in Neural Information Processing Systems 29, Barcelona SPAIN, Jul. 2016, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1607.06450.
[21] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet Allocation,” J. Mach. Learn. Res., vol. 3, no. Jan, pp. 993–1022, 2003.
[22] G. E. Hinton, “Learning distributed representations of concepts,” Amherst, Massachusetts, 1986, vol. 1, pp. 1–12.
[23] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A Neural Probabilistic Language Model,” J. Mach. Learn. Res., vol. 3, no. Feb, pp. 1137–1155, 2003.
[24] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” Scottsdale, AZ, USA, Jan. 2013, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1301.3781.
[25] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed Representations of Words and Phrases and their Compositionality,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 3111–3119.
[26] J. Pennington, R. Socher, and C. Manning, “GloVe: Global Vectors for Word Representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, Oct. 2014, pp. 1532–1543, doi: 10.3115/v1/D14-1162.
[27] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of Tricks for Efficient Text Classification,” in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain, Apr. 2017, pp. 427–431, Accessed: Sep. 21, 2020. [Online]. Available: https://www.aclweb.org/anthology/E17-2068.
[28] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching Word Vectors with Subword Information,” Trans. Assoc. Comput. Linguist., vol. 5, pp. 135–146, 2017, doi: 10.1162/tacl_a_00051.
[29] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov, “Learning Word Vectors for 157 Languages,” presented at the LREC 2018, Miyazaki, Japan, May 2018, Accessed: Sep. 21, 2020. [Online]. Available: https://www.aclweb.org/anthology/L18-1550.
[30] S. Junyi, fxsjy/jieba. 2020.
[31] S. Ruder, “An Overview of Multi-Task Learning in Deep Neural Networks,” ArXiv170605098 Cs Stat, Jun. 2017, Accessed: Sep. 21, 2020. [Online]. Available: http://arxiv.org/abs/1706.05098.
[32] A. M. Dai and Q. V. Le, “Semi-supervised Sequence Learning,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, Cambridge, MA, USA, Dec. 2015, pp. 3079–3087, Accessed: Sep. 21, 2020. [Online].
[33] Zeng, Chenglin, Laizhong Cui, and Zhi Wang. "An Exponential Time-Aware Recommendation Model for Mobile Notification Services." Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, Cham, 2017.
[34] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognit. Lett., vol. 27, no. 8, pp. 861–874, Jun. 2006, doi: 10.1016/j.patrec.2005.10.010.
[35] Y. Sasaki, “The truth of the F-measure,” p. 5, Oct. 2007.
[36] J.-W. Li et al., “Interactome-transcriptome analysis discovers signatures complementary to GWAS Loci of Type 2 Diabetes,” Sci. Rep., vol. 6, p. 35228, Oct. 2016, doi: 10.1038/srep35228.
[37] Le, Quoc, and Tomas Mikolov. "Distributed representations of sentences and documents." International conference on machine learning. 2014.
[38] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018).