我們生活的一切都與「時間」這個重要的元素息息相關，透過時間，我們可以將生活的許多事物都稱之為序列。例如，打卡的歷史記錄，一種按時間排序排列的序列數據。隨著Facebook和Twitter這些社交網絡的快速發展，越來越多的時空數據被收集和研究。因此，預測使用者的下一個打卡位置變得至關重要。儘管有大量文獻研究提出特定的方法來解決此問題，但它們仍存有一些缺陷。例如嵌套式長短期記憶模型(Nested LSTM)，其缺點是它將每一個時間點的打卡記錄都視為相同的權重，然而隨著打卡的歷史記錄變長，Nested LSTM可能會被誤導，並且無法在打卡的歷史記錄中找到使用者的行為規律，進一步降低了預測使用者在下一個地點打卡的準確度。在本文中，我們擴展了Nested LSTM並提出了基於注意力機制之分層長短期記憶網路(Hierarchical LSTM with Attention Module)的新方法，該方法不僅使用Hierarchical LSTM來處理打卡類別的分層結構，並且還應用注意機制來提升預測的準確度。實驗結果顯示，本文提出的Hierarchical LSTM with Attention Module，在Accuracy@5平均提高了12.78 ％和10.22 ％分別對應於Foursquare和Jiepang這兩個資料庫上。

Everything in life depends on time and therefore, represents a sequence. For example, check-in history is one kind of sequential data sorted by time. With the rapid growth of popular social networks such as Facebook and Twitter, more and more temporal and spatial contexts have been collected and studied. Hence predicting the intention of a user's next check-in location becomes crucial and achievable. Some works have been proposed to address this problem, but they all have their defects. Nested LSTM is constructed based on viewing every check-in sequence as the same weight. Along with a check-in sequence going long, Nested LSTM might be misleading and couldn't find a pattern in a check-in history. In this thesis, we extend Nested LSTM and propose a novel method called Hierarchical LSTM with Attention Module that not only uses the Hierarchical LSTM to deals with the taxonomy structure of venues but also applies the attention mechanism to provide accurate predictions on the category of a user's next check-in location. Experimental results show that Hierarchical LSTM with Attention Module proposed in this thesis improves Accuracy@5 by 12.78\% and 10.22\% on Foursquare and Jiepang on average, respectively.

1 Introduction------------------------------------------1
2 Preliminaries-----------------------------------------5
2.1 Datasets: Foursquare & Jiepang---------------------5
2.2 Observations & Assumptions-------------------------6
3 Related Work------------------------------------------8
3.1 Conventional Approaches----------------------------8
3.2 Neural-based Approaches----------------------------9
4 Proposed Model---------------------------------------11
4.1 Problem Formulation-------------------------------11
4.2 Long Short-Term Memory----------------------------12
4.3 Hierarchical LSTM with Attention Module-----------14
4.4 Simulate The Real-word Situation------------------16
5 Experimental Results---------------------------------19
5.1 Experimental Setup--------------------------------19
5.2 Result Summary------------------------------------21
5.3 Delta Time Analysis-------------------------------25
6 Conclusions------------------------------------------27
References---------------------------------------------28

[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate, 2014. cite arxiv:1409.0473Comment: Accepted at ICLR 2015 as oral presentation.
[2] J. Bjerva, J. Bos, R. Van der Goot, and M. Nissim. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval2014), pages 642–646, 2014.
[3] J. Chang and E. Sun. Location3: How users share and respond to location-based data on social. In ICWSM, 2011.
[4] C. Cheng, H. Yang, M. R. Lyu, and I. King. Where you like to go next: Successive point-of-interest recommendation. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI ’13, pages 2605–2611. AAAI Press, 2013.
[5] H. Gao, J. Tang, and H. Liu. Exploring social-historical ties on location-based social networks. In ICWSM, 2012.
[6] J. L. Herlocker, J. A. Konstan, A. Borchers, and J. Riedl. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22Nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’99, pages 230–237. ACM, 1999.
[7] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[8] Y. Koren, R. Bell, and C. Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37, Aug. 2009.
[9] D. Lian and X. Xie. Collaborative activity recognition via check-in history. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks, pages 45–48. ACM, 2011.
[10] Q. Liu, S. Wu, L. Wang, and T. Tan. Predicting the next location: A recurrent model with spatial and temporal contexts. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 194–200. AAAI Press, 2016.
[11] T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics.
[12] T. Mikolov, M. Karafit, L. Burget, J. Cernock, and S. Khudanpur. Recurrent neural network based language model. In T. Kobayashi, K. Hirose, and S. Nakamura, editors, INTERSPEECH, pages 1045–1048. ISCA, 2010.
[13] A. Noulas, C. Mascolo, and E. Frias-Martinez. Exploiting foursquare and cellular data to infer user activity in urban environments. In Mobile Data Management (MDM), 2013 IEEE 14th International Conference on, volume 1, pages 167–176. IEEE, 2013.
[14] H. Palangi, L. Deng, Y. Shen, J. Gao, X. He, J. Chen, X. Song, and R. Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 24(4):694–707, Apr. 2016.
[15] F. Pianese, X. An, F. Kawsar, and H. Ishizuka. Discovering and predicting user routines by differential analysis of social network traces. In World of Wireless, Mobile and Multimedia Networks (WoWMoM), 2013 IEEE 14th International Symposium and Workshops on a, pages 1–9. IEEE, 2013.
[16] S. Rendle, C. Freudenthaler, and L. Schmidt-Thieme. Factorizing personalized markov chains for next-basket recommendation. pages 811–820, 01 2010.
[17] R. Salakhutdinov and A. Mnih. Probabilistic matrix factorization. In Proceedings of the 20th International Conference on Neural Information Processing Systems, NIPS’07, pages 1257–1264, USA, 2007. Curran Associates Inc.
[18] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web, WWW ’01, pages 285–295. ACM, 2001.
[19] R. Socher, C. C. Lin, C. Manning, and A. Y. Ng. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 129–136, 2011.
[20] X.-A. Tseng. Nested lstm : Modeling taxonomy and temporal dynamics in locationbased social network. 2018.
[21] P. Wang, J. Guo, Y. Lan, J. Xu, S. Wan, and X. Cheng. Learning hierarchical representation model for nextbasket recommendation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’15, pages 403–412, New York, NY, USA, 2015. ACM.
[22] C.-Y. Wu, A. Ahmed, A. Beutel, A. J. Smola, and H. Jing. Recurrent recommender networks. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 495–503, 2017.
[23] L. Xiong, X. Chen, T.-K. Huang, J. G. Schneider, and J. G. Carbonell. Temporal collaborative filtering with bayesian probabilistic tensor factorization. In SDM, 2010.
[24] J. Yang, J. Xu, M. Xu, N. Zheng, and Y. Chen. Predicting next location using a variable order markov model. In Proceedings of the 5th ACM SIGSPATIAL International Workshop on GeoStreaming, IWGS ’14, pages 37–42, New York, NY, USA, 2014. ACM.
[25] J. Ye, Z. Zhu, and H. Cheng. What’s your next move: User activity prediction in location-based social networks. In Proceedings of the 2013 SIAM International Conference on Data Mining, pages 171–179. SIAM, 2013.
[26] Q. Yuan, G. Cong, Z. Ma, A. Sun, and N. M. Thalmann. Time-aware point-ofinterest recommendation. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, pages 363–372. ACM, 2013.
[27] Y. Zhang, H. Dai, C. Xu, J. Feng, T. Wang, J. Bian, B. Wang, and T.-Y. Liu. Sequential click prediction for sponsored search with recurrent neural networks. In AAAI, volume 14, pages 1369–1375, 2014.
[28] Z. Zhang, C. Li, Z. Wu, A. Sun, D. Ye, and X. Luo. NEXT: A neural network framework for next POI recommendation. CoRR, abs/1704.04576, 2017.
[29] Y. Zhu, H. Li, Y. Liao, B. Wang, Z. Guan, H. Liu, and D. Cai. What to do next: Modeling user behaviors by time-lstm. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, pages 3602–3608. AAAI Press, 2017.