隨著科技的發展，多媒體的資訊量與日俱增，成為一種傳遞訊息與想法的重要通訊管道，對個人與社會帶來各方面的影響，個人而言，在接收端的我們因為不同的人格特質、喜好，可能誘發出不同的情緒感受；對社會而言，多媒體可以創造商業價值，或用來增加社會議題的討論，因此能同時了解多媒體富含的意義與受眾的反應，更能達到多媒體傳輸的目的，人工智慧技術的興起，使得我們可以多面向、量化的討論多媒體與個人、社會的關係。
在此篇研究中我們探討觀看影片時受眾的即時反應，在兩個實驗中分別以社群與個人的角度進行研究，實驗一利用電影預告的內容與受眾觀看預告時臉部表情的反應建立電影票房預測模型，並提出基於電影種類的鑲嵌表徵學習架構（Intra-genre projection）以提升票房預測準確率。實驗二我們更深入研究影片內容與觀眾反應的關係，考慮到個人的差異，建立個人化的情緒反應預測模型，並結合推薦系統(Recommendation System)的概念，運用基於影片資訊的方法與實驗一提出的種類鑲嵌表徵學習架構，在即使只有一個人少量的資訊下驗證可行性。總結實驗一與實驗二可以發現人的反應因為影片的類型而有差異，且在影片的設計上為了激發不同的情緒，各類型的影片會有不同的特徵，因此透過以類型為基礎的鑲嵌表徵學習架構，先區分出不同的類型，再進行票房或情緒反應可以增進預測準確率。

With the improvement in technology, multimedia contents increase rapidly. Multimedia is a communication medium for people convey certain concepts or emotion, and effects individuals and society. For individuals, we experience certain emotion or feelings while interacting with media contents. Induced feelings and emotion are often varying due to individual difference. As for society, multimedia makes impacts to economics and politics. It can also make some social issues bring to public attention. As the rise of artificial intelligence, people use machine learning methods to analyze large-scale and comprehensive multimedia content and human behaviors.
In this work, we learn people’s just-in-time reaction while watching video clips. We design two experiments for learning the impact of multimedia on society and individuals respectively. In the first experiment, we utilize trailers content and audience reaction to build a box office prediction model. We propose a novel intra-genre embedding which project raw features to another latent space considering the genre to improve predictive power. In the second experiment, we further investigate the relation between audience and content. Considering individual differences, we build personalized models. Their natural reaction during video watching can be regarded as certain ratings or preference. Motivated by the concept of Recommendation System, we propose content-based methods using the similar genre-based projection in experiment 1 to learn personalized reaction and deal with lack of data problems. Summarizing experiment 1 and experiment 2, we find that genre plays an important role in shaping the trailer content and viewer response. By appropriately projecting content and expression features onto a minimal intra-genre representation space, it effectively mitigates the unwanted variability in the original feature space and, hence, enhances their discriminatory power in box office prediction and personalized reaction prediction.

摘要....................................................I
ABSTRACT...............................................II
誌謝....................................................IV
CONTENTS................................................V
CHAPTER 1 INTRODUCTION..................................1
1.1 RELATED WORK.......................................3
CHAPTER 2 DATABASE: REACTION VIDEOS.....................6
CHAPTER 3 EXPERIMENT1: BOX OFFICE PREDICTION USING TRAILER CONTENT AND VIEWERS’ REACTION...................................8
3.1 METHODOLOGY........................................8
3.1.1 Dataset: Movie Trailer and Reactors.............8
3.1.2 Features........................................9
3.1.3 Network Architecture: Intra-Genre Projection...14
3.2 EXPERIMENTS & RESULTS.............................16
3.3 DISCUSSION & ANALYSIS.............................18
CHAPTER 4 EXPERIMENT2: PERSONALIZED REACTION MODEL.....20
4.1 DATASET...........................................20
4.1.1 Features Extraction and Label Setting..........22
4.2 METHODOLOGY.......................................24
4.2.1 Deep Neural Network (DNN)......................25
4.2.2 Recommender System Model.......................25
4.2.3 High Order SVD (HOSVD).........................26
4.2.4 Deep Matrix Factorization (DMF)................28
4.2.5 Deep Variational Matrix Factorization (VDMF)...29
4.3 EXPERIMENTS & RESULTS.............................31
4.4 DISCUSSION & ANALYSIS.............................33
4.4.1 Model Comparison...............................33
4.4.2 Person and Genre...............................34
4.4.3 Content and Genre..............................35
CHAPTER 5 CONCLUSION...................................38
REFERENCE..............................................40

[1] Anne Bartsch, Peter Vorderer, Roland Mangold and Reinhold Viehoff, “Appraisal of Emotions in Media Use: Toward a Process Model of Meta-Emotion and Emotion Regulation” in Media Psychology, vol. 11, no. 1, pp. 7-27, 2008
[2] Adelaar, Thomas, et al. "Effects of media formats on emotions and impulse buying intent." in Journal of Information Technology, 18.4, pp. 247-266, 2003
[3] Doyle, Gillian. “Understanding media economics.” SAGE Publications Limited, 2013.
[4] Gitlin, Todd. "Media sociology." Theory and society 6.2 (1978): 205-253.
[5] Y. Lin et al., "EEG-Based Emotion Recognition in Music Listening," in IEEE Transactions on Biomedical Engineering, vol. 57, no. 7, pp. 1798-1806, July 2010.
[6] M. Soleymani, M. Pantic and T. Pun, "Multimodal Emotion Recognition in Response to Videos," in IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 211-223, April-June 2012
[7] Douiji yasmina, Mousannif Hajar, Al Moatassime Hassan, “Using YouTube Comments for Text-based Emotion Recognition,” in Procedia Computer Science, vol. 83, pp. 292-299, 2016,
[8] Deng, Zhiwei, et al. "Factorized variational autoencoders for modeling audience reactions to movies." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[9] Vonderschmitt, Kaitlin. "The growing use of social media in political campaigns: How to use Facebook, Twitter and YouTube to create an effective social media campaign." (2012).
[10] Wühr, Peter, Benjamin P. Lange, and Sascha Schwarz. "Tears or fears? Comparing gender stereotypes about movie preferences to actual preferences." Frontiers in psychology 8 (2017): 428.
[11] Saeed, Henna, et al. "Gazed representation: Analysis of gender portrayal in Hindi and English music videos." Journal of Media Studies 28.2 (2019).
[12] Jeffrey S. Simonoff and Ilana R. Sparrow, “Predicting movie grosses: Winners and losers, blockbusters and sleepers,” CHANCE, vol. 13, no. 3, pp. 15–24, 2000. [13] Byeng-Hee Chang and Eyun-Jung Ki, “Devising a practical model for predicting theatrical movie success: Focusing on the experience good property,” Journal of Media Economics, vol. 18, no. 4, pp. 247–269, 2005.
[14] Ramesh Sharda and Dursun Delen, “Predicting boxoffice success of motion pictures with neural networks,” Expert Systems with Applications, vol. 30, no. 2, pp. 243–254, 2006.
[15] T. G. Rhee and F. Zulkernine, “Predicting movie box office profitability: A neural network approach,” in 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), Dec 2016, pp. 665–670.
[16] Y. Hou, T. Xiao, S. Zhang, X. Jiang, X. Li, X. Hu, J. Han, L. Guo, L. S. Miller, R. Neupert, and T. Liu, “Predicting movie trailer viewer’s “like/dislike” via learned shot editing patterns,” IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 29–44, Jan 2016
[17] A. Tadimari, N. Kumar, T. Guha, and S. S. Narayanan,“Opening big in box office? trailer content can help,” in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), 2016, pp. 2777–2781.
[18] Gilad Mishne and Natalie Glance, “Predicting movie sales from blogger sentiment,” AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp. 155–158, Jan 2006.
[19] Andrei Oghina, Mathias Breuss, Manos Tsagkias, and Maarten de Rijke, “Predicting imdb movie ratings using social media,” in Advances in Information Retrieval, Berlin, Heidelberg, 2012, pp. 503–507, Springer Berlin Heidelberg.
[20] S. Asur and B. A. Huberman, “Predicting the future with social media,” in 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Aug 2010, vol. 1, pp. 492–499.
[21] M´arton Mesty´an, Taha Yasseri, and J´anos Kert´esz, “Early prediction of movie box office success based on wikipedia activity big data,” PLOS ONE, vol. 8, no. 8, pp. 1–8, Aug 2013.
[22] Hee Lin Wang and Loong-Fah Cheong, “Affective understanding in film,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, no. 6, pp. 689– 704, June 2006.
[23] Hamann, Stephan, and Turhan Canli. "Individual differences in emotion processing." Current opinion in neurobiology 14.2 (2004): 233-238.
[24] Norman, Jonathan. "PERSONALITY TYPES AND THE ENJOYMENT OF HORROR MOVIES." Journal of Social & Psychological Sciences 11.1 (2018).
[25] Zhao, Sicheng, et al. "Predicting personalized emotion perceptions of social images." Proceedings of the 24th ACM international conference on Multimedia. 2016.
[26] Yeh, Chan-Chang, et al. "Building a personalized music emotion prediction system." Pacific-Rim Conference on Multimedia. Springer, Berlin, Heidelberg, 2006.
[27] Zhao, Sicheng, et al. "Personality-Aware Personalized Emotion Recognition from Physiological Signals." IJCAI. 2018.
[28] T. Guha, N. Kumar, S. S. Narayanan, and S. L. Smith, “Computationally deconstructing movie narratives: An informatics approach,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr 2015, pp. 2264–2268.
[29] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. Andr´e, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. 190–202, Apr 2016.
[30] Florian Eyben, Martin W¨ollmer, and Bj¨orn Schuller, “opensmile – the munich versatile and fast open-source audio feature extractor,” in Proc. ACM Multimedia (MM), Oct 2010, pp. 1459–1462.
[31] Patricia Valdez and Albert Mehrabian, “Effects of color on emotions,” Journal of Experimental Psychology, vol.123, no. 4, pp. 394–409, 1994.
[32] H. Wang, A. Kl¨aser, C. Schmid, and C. Liu, “Action recognition by dense trajectories,” in CVPR 2011, Jun 2011, pp. 3169–3176.
[33] Jorge Sanchez, Florent Perronnin, Thomas Mensinka, and Jakob Verbeek, “Image classification with the fisher vector: Theory and practice,” International Journal of Computer Vision, vol. 105, no. 3, pp. 222–245, Dec 2013.
[34] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), May 2018, pp.59–66.
[35] Zhang, Ying, and Huchuan Lu. "Deep cross-modal projection learning for image-text matching." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[36] Lucey, Patrick, et al. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression." 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE, 2010.
[37] Hug, Nicolas, "Surprise, a Python library for recommender systems", http://surpriselib.com, 2017
[38] Mnih, Andriy, and Russ R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008.
[39] S. Funk, “Netflix Update: Try This At Home”, http://sifter.org/˜simon/journal/20061211.html, 2006.
[40] Koren, Yehuda. "Factorization meets the neighborhood: a multifaceted collaborative filtering model." Proceedings of the ACM SIGKDD international conference on Knowledge discovery and data mining. 2008.
[41] Karatzoglou, Alexandros, et al. "Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering." Proceedings of the fourth ACM conference on Recommender systems. 2010.
[42] Lian, Defu, et al. "Regularized content-aware tensor factorization meets temporal-aware location recommendation." 2016 IEEE 16th international conference on data mining (ICDM). IEEE, 2016.
[43] Xue, Hong-Jian, et al. "Deep Matrix Factorization Models for Recommender Systems." IJCAI. Vol. 17. 2017.
[44] Yi, Baolin, et al. "Deep matrix factorization with implicit feedback embedding for recommendation system." IEEE Transactions on Industrial Informatics 15.8 (2019): 4591-4601.
[45] Shen, Xiaoxuan, et al. "Deep Variational Matrix Factorization with Knowledge Embedding for Recommendation System." IEEE Transactions on Knowledge and Data Engineering (2019).
[46] Ming-Ya Ko, Jeng-Lin Li, Chi-Chun Lee, "Learning Minimal Intra-Genre Multimodal Embedding from Trailer Content and Reactor Expressions for Box Office Prediction", in Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), 2019