群眾外包提供了人們一個線上的平台去收集大量的人工標記進而解決相對複雜的任務。然而，雇用的工作者可能因為非專長或者是個人的因素導致回答的標記包含了雜訊。因此，在收集到這些包含雜訊的標記時，我們必須判斷每位工作者的特長進而判斷提供的回答標記是否可靠。本篇論文探討了兩個關於如何在群眾外包環境下聚合工作者標籤的主題。第一個主題延伸了已提出的排名聚合算法 SpecRank [1]，該演算法利用了隱性特徵建構了一個模型能夠在群眾外包環境中判斷具有不同專業的工作者提供的成對比較。通過採用貝葉斯方法(Bayesian method)，所延伸提出的演算法提高了計算效率並使模型更加穩健。第二個主題整合出了一套基於情緒反應辨識及生理訊號偵測技術建構下世代多媒體推薦系統。主要針對多媒體觀影者反應資料庫建構情緒及行為模型，同時計算臉部表情、呼吸、心跳等多模態統計數值，加以電影種類群內最小化投影學習整合觀影者表情及內容之表徵，用於推薦系統之機器學習模型並融合非侵入式行為分析。此模型可避免預測結果受到使用者對不同電影類別偏好之評價偏差所造成的影響。透過上述分析，本系統透過整合電影內涵與情緒及生理反應的關係量化影片受歡迎程度。

Crowdsourcing provides an online environment for those who try to solve large-scale tasks that require human intelligence. However, the hired workers sometimes are of low quality or even malicious. For this reason, detecting workers' specialty to obtain labels without noise from workers is important. This work includes two topics on how to aggregate labels from workers under a crowdsourcing setting. The first topic extends the proposed rank aggregation algorithm in \cite{hong2021specrank}, SpecRank, from pairwise comparisons provided by workers with different specialties in a crowdsourced setting. By adopting the Bayesian approach, the proposed method improves the efficiency of computation and makes the model much more general. The second topic develops a framework that integrates affective responses and contactless behavioral modeling technologies in a personalized multimedia recommendation system. When users are watching videos, their reaction is recorded by a camera and physiological device. Then, the movie contents and facial reactions are learned as representations using a deep intra-genre projection network. The proposed system finally predicts users' preferences about multimedia content, including movies, music, TV shows, etc. In addition, we utilize the proposed system to collect a dataset and analyze the relationship between personal preference and affective response.

Contents
1 Introduction......................... 1
2 Background and Related Works......................... 4
2.1 Related Works on Rank Aggregation......................... 4
2.2 Related Works on Recommendation System with Affective Content........ 6
3 Bayesian SpecRank......................... 8
3.1 Variational Approximation.............................. 11
3.2 Bayesian SpecRank Parameter Estimation via the Variational EM Algorithm.... 14
4 Multimedia Recommendation using Viewer Affective Behavior Responses......................... 30
4.1 Demo System..................................... 30
4.2 Experimental Setup.................................. 32
4.2.1 Experimental scenarios............................ 32
4.2.2 Video Selection and Recordings....................... 33
4.2.3 Physiological Signals and Instruments.................... 33
4.2.4 Display Platform............................... 33
4.3 Database Introduction................................. 34
4.4 Recommendation on Database............................ 35
5 Experimental Results on Ranking Aggregation......................... 38
5.1 Baseline Methods and Evaluation Criteria...................... 38
5.2 Evaluation Metric................................... 39
5.3 Initialization...................................... 40
5.4 Last.fm Dataset.................................... 41
5.5 Journal Dataset.................................... 42
5.6 Movie Dataset..................................... 45
6 Conclusion......................... 47
Bibliography......................... 48

[1] Y.-W. P. Hong, W. Y. Chen, and H. T. Chang, “Specrank: Ranking by pairwise comparisons from specialty workers in a crowdsourced setting,” in IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2021.
[2] “Amazon Mechanical Turk,” 2001. https://www.mturk.com/.
[3] “Figure Eight,” 2007. https://www.figure-eight.com.
[4] “Clickworker,” 2009. https://www.clickworker.com.
[5] R. M. Borromeo and M. Toyama, “Automatic vs. crowdsourced sentiment analysis,” in Proceedings of the 19th International Database Engineering & Applications Symposium, pp. 90–95, 2015.
[6] G. Xintong, W. Hongzhi, Y. Song, and G. Hong, “Brief survey of crowdsourcing for data mining,” Expert Systems with Applications, vol. 41, no. 17, pp. 7987 – 7994, 2014.
[7] A. Kovashka, O. Russakovsky, L. Fei-Fei, K. Grauman, et al., “Crowdsourcing in computer vision,” Foundations and Trends® in computer graphics and Vision, vol. 10, no. 3, pp. 177–243, 2016.
[8] C. E. Izard, The psychology of emotions. Springer Science & Business Media, 1991.
[9] R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952.
[10] S. Negahban, S. Oh, and D. Shah, “Rank Centrality: Ranking from pairwise comparisons,”
Operations Research, vol. 65, pp. 266–287, 2016.
[11] X. Chen, P. N. Bennett, K. Collins-Thompson, and E. Horvitz, “Pairwise ranking aggregation in a crowdsourced setting,” in Proceedings of the sixth ACM international conference on Web
search and data mining, pp. 193–202, 2013.
[12] N. Bugakova, V. Fedorova, G. Gusev, and A. Drutsa, “Aggregation of pairwise comparisons with reduction of biases,” arXiv preprint arXiv:1906.03711, 2019.
[13] Y. Sun, J. Han, P. Zhao, Z. Yin, H. Cheng, and T.Wu, “RankClus: Integrating clustering with ranking for heterogeneous information network analysis,” in Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology (EDBT), pp. 565–576, 2009.
[14] T. S. Jaakkola and M. I. Jordan, “Bayesian parameter
estimation via variational methods,”
Statistics and Computing, pp. 25–37, 2000.
[15] G. Linden, B. Smith, and J. York, “Amazon.com recommendations: item-to-item collaborative filtering,” IEEE Internet Computing, vol. 7, no. 1, pp. 76–80, 2003.
[16] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-based collaborative filtering recommendation algorithms,” in Proceedings of the 10th international conference on World Wide Web, pp. 285–295, 2001.
[17] Y. Koren, R. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” Computer, vol. 42, no. 8, pp. 30–37, 2009.
[18] D. Ayata, Y. Yaslan, and M. E. Kamasak, “Emotion based music recommendation system using wearable physiological sensors,” IEEE Transactions on Consumer Electronics, vol. 64, no. 2, pp. 196–203, 2018.
[19] Y. Moshfeghi, B. Piwowarski, and J. M. Jose, Handling Data Sparsity in Collaborative Filtering Using Emotion and Semantic Based Features. Association for Computing Machinery, 2011.
[20] S. Theodoridis, Machine Learning: a Bayesian and Optimization Perspective. Academic Press, 2015.
[21] J. Li, Y. Baba, and H. Kashima, “Simultaneous clustering and ranking from pairwise comparisons,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pp. 1554–1560, 2018.
[22] T. T. Nguyen, F. M. Harper, L. Terveen, and J. A. Konstan, “User personality and user satisfaction with recommender systems,” Information Systems Frontiers, vol. 20, no. 6, pp. 1173–1189, 2018.
[23] W. R. Gilks, A. Thomas, and D. J. Spiegelhalter, “A language and program for complex bayesian modelling,” Journal of the Royal Statistical Society: Series D (The Statistician), vol. 43, no. 1, pp. 169–177, 1994.
[24] C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan, “An introduction to mcmc for machine
learning,” Machine learning, vol. 50, no. 1-2, pp. 5–43, 2003.
[25] D. J. Spiegelhalter and S. L. Lauritzen, “Sequential updating of conditional probabilities on directed graphical structures,” Networks, vol. 20, no. 5, pp. 579–605, 1990.
[26] R. Ekman, What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA, 1997.
[27] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59–66, IEEE, 2018.
[28] “Texas instruments mmwave sensors.,” 1995. https://www.ti.com/sensing-products/mmwave/overview.html.
[29] I. Cantador, P. Brusilovsky, and T. Kuflik, “2nd workshop on information heterogeneity and fusion in recommender systems (hetrec 2011),” in Proceedings of the 5th ACM conference on Recommender systems, RecSys 2011, 2011.
[30] U. Von Luxburg, “A tutorial on spectral clustering,” Statistics and computing, vol. 17, no. 4, pp. 395–416, 2007.
[31] P. Diaconis, “Group representations in probability and statistics,” Lecture notes-monograph series, vol. 11, pp. i–192, 1988.
[32] M. G. Kendall, “Rank correlation methods.,” Journal of the Institute of Actuaries, 1948.