欺騙檢測在法學、商業、心理學等許多領域中一直是熱門的議題。近期隨著機器學習與計 算機視覺的發展，傳統欺騙檢測一部分的研究重心逐漸轉向到與自動辨識結合的視頻欺騙 檢測。科技與舊議題的結合固然令人振奮，但是仍有許多問題值得進一步研究，其中主要 挑戰之一是數據短缺問題。截至目前為止，僅發布了一個有關欺騙檢測的多模態基準數據 集，其中包含 121 個用於欺騙檢測的視頻剪輯 (欺騙性剪輯佔 61 部，真性剪輯佔 60 部)。 因此，對於這個訓練資料為數不多的資料集，大多數生成的欺騙檢測模型 (尤其是基於深 度神經網絡的方法) 都存在過擬合問題導致泛化能力不足。為了解決這些問題，我們提出 了一種新穎的情感轉換特徵 (ETF) 來分析有限數據下的欺騙檢測。對所提出的方法與最新 的多模態方法進行的分析和比較，結果表明識別效能可達到 91.67 準確率% 和 0.92 AUC 值。

Deception detection has always been a hot topic in many fields such as law, business, psychology and so on. Recently, with the development of machine learning and computer vision, part of the research focus of traditional deception detection has gradually turned to deception detection in video combined with automatic recognition. The combination of technology and old issues is exciting, but there are still many issues that deserve further study. One of the main challenges is the data shortage. So far, only one multi­modal benchmark dataset on deception detection has been published, which contains 121 video clips for deception detection (61 deceptive clips and 60 truthful clips). Due to the lack of training data available from the data set, most of the generated fraud detection models (especially based on deep neural network methods) have overfitting problems, resulting in poor generalization ability. In order to solve these problems, we proposed a novel emotional transformation feature (ETF) to analyze deception detection under limited data. The analysis and comparison between the proposed method and the multimodal method show that the recognition efficiency can reach an accuracy of 91.67 % and AUC value of 0.92

1 緒論 ........................................................................................... 1
1.1 動機與目的.............................................................................. 1
1.2 論文架構 ................................................................................ 2
2 相關研究探討 ................................................................................ 3
2.1 視覺模態 ................................................................................ 3
2.2 聲學模態 ................................................................................ 4
2.3 語言模態 ................................................................................ 4
3 系統架構 ...................................................................................... 5
3.1 多模態欺騙辨識架構 .................................................................. 5
3.2 情緒轉變特徵 (Emotional Transformation Feature) .................................. 6
3.3 結合聲音資訊修正情緒轉變特徵..................................................... 8
3.4 分類器 ................................................................................... 10
3.4.1 決策樹 (Decision Tree) ......................................................... 10
3.4.2 隨機森林 (Random Forest)..................................................... 12
3.4.3 k­近鄰演算法 (k­Nearest Neighbors) ......................................... 14
3.4.4 支援向量機 (Support Vector Machine)........................................ 15
4 實驗與結果 ................................................................................... 18
4.1 資料集 ................................................................................... 18
4.2 實驗方法與評估指標 .................................................................. 19
4.3 參數設定與實驗結果 .................................................................. 21
iii5 結論與未來展望.............................................................................. 24
參考文獻......................................................................................... 25

[1] K. Serota, T. Levine, and F. Boster, “The prevalence of lying in america: Three studies of
self‐reported lies,” Human Commun. Res., vol. 36, pp. 2–25, Dec. 2009.
[2] R. S. Feldman, J. A. Forrest, and B. R. Happ, “Self­presentation and verbal deception: Do
self­presenters lie more?” Basic and Appl. Social Psychol., vol. 24, no. 2, pp. 163–170, 2002.
[3] C. Bond and B. DePaulo, “Accuracy of deception judgments,” Personality and Social Psychol. Rev., vol. 10, pp. 214–234, Feb. 2006.
[4] National Research Council, “The polygraph and lie detection,” The National Academies
Press, 2003.
[5] M. Farah, J. Hutchinson, E. Phelps, and A. Wagner, “Functional MRI­based lie detection:
Scientific and societal challenges,” Nature Rev. Neuroscience, vol. 15, pp. 123–131, Feb.
2014.
[6] V. Pérez­Rosas, M. Abouelenien, R. Mihalcea, and M. Burzo, “Deception detection using
real­life trial data,” in 2015 Proc. ACM Int. Conf. Multimodal Interact., Seattle, USA, Nov.
2015, pp. 59–66.
[7] M. Gogate, A. Adeel, and A. Hussain, “Deep learning driven multimodal fusion for automated
deception detection,” in 2017 Proc. IEEE Symp. Ser. Comput. Intell., Nov.­Dec. 2017, pp. 1–
6.
[8] G. Krishnamurthy, N. Majumder, S. Poria, and E. Cambria, “A deep learning approach for
multimodal deception detection,” arXiv preprint arXiv:1803.00344, Mar. 2018.
[9] F. A. Al­Smadi, “Detection of deceptive behavior: A cross­cultural test,” Social Behav. and
Personality, vol. 28, no. 5, pp. 455–462, Oct. 2000.
[10] C. Bond, A. Atoum, A. Mahmoud, and R. Bonser, “Lie detection across cultures,” J. Nonverbal Behav., vol. 14, no. 3, pp. 189–204, 1990.
[11] M. Burzo, M. Abouelenien, V. Perez­Rosas, and R. Mihalcea, Multimodal Deception Detection. Association for Computing Machinery and Morgan & Claypool, 2018, pp. 419–453.
[12] P. Ekman, Telling Lies: Clues to Deceit in the Marketplace, Politics and Marriage. Norton,
W.W. and Company, 2001.
[13] ——, “Darwin, deception, and facial expression,” Ann. New York Acad. Sciences, vol. 1000,
pp. 205–221, Jan. 2004.
[14] Y.­L. Tian, T. Kanade, and J. F. Cohn, Facial Expression Analysis. Springer, New York,
NY, 2001, pp. 247–275.
[15] M. Owayjan, A. Kashour, N. Al Haddad, M. Fadel, and G. Al Souki, “The design and development of a lie detection system using facial micro­expressions,” in 2012 Proc. 2nd Int.
Conf. Advances Comput. Tools for Eng. Appl., 2012, pp. 33–38.
[16] S. Lu, G. Tsechpenakis, D. Metaxas, M. Jensen, and J. Kruse, “Blob analysis of the head
and hands: A method for deception detection,” in Proc. 38th Annu. Hawaii Int. Conf. Syst.
Sciences, Feb. 2005, pp. 20–29.
[17] T. O. Meservy, M. L. Jensen, J. Kruse, J. K. Burgoon, J. F. Nunamaker, D. P. Twitchell,
G. Tsechpenakis, and D. N. Metaxas, “Deception detection through automatic, unobtrusive
analysis of nonverbal behavior,” IEEE Intell. Syst., vol. 20, no. 5, pp. 36–43, Sept. 2005.
[18] L. Caso, F. Maricchiolo, M. Bonaiuto, A. Vrij, and S. Mann, “The impact of deception and
suspicion on different hand movements,” J. Nonverbal Behav., vol. 30, pp. 1–19, Feb. 2006.
[19] D. Cohen, G. Beattie, and H. Shovelton, “Nonverbal indicators of deception: How iconic
gestures reveal thoughts that cannot be suppressed,” Semiotica, vol. 182, pp. 133–174, Oct.
2010.
[20] J. Hillman, A. Vrij, and S. Mann, “Um … they were wearing … : The effect of deception on
specific hand gestures,” Legal and Criminological Psychology, vol. 17, pp. 336–345, Sept.
2012.
[21] H. Karimi, J. Tang, and Y. Li, “Toward end­to­end deception detection in videos,” in 2018
Proc. IEEE Int. Conf. Big Data, Dec. 2018, pp. 1278–1283.
[22] P. Ekman, M. OŚullivan, W. V. Friesen, and K. R. Scherer, “Invited article: Face, voice, and
body in detecting deceit,” J. Nonverbal Behav., vol. 15, no. 2, pp. 125–135, Dec. 1991.
[23] S. Benus, F. Enos, J. Hirschberg, and E. Shriberg, “Pauses in deceptive speech,” in Proc. Int.
Symp. Comput. Architecture, 2006, pp. 1–4.
[24] B. DePaulo, J. J. Lindsay, B. Malone, L. Muhlenbruck, K. Charlton, and H. Cooper, “Cues
to deception,” Psychological Bull., vol. 129, no. 1, pp. 74–118, Feb. 2003.
[25] J. Hirschberg, S. Benus, J. Brenier, F. Enos, S. Friedman, S. Gilman, C. Girand, M. Graciarena, and A. Kathol, “Distinguishing deceptive from non­deceptive speech,” in Proc. Ann.
Conf. Int. Speech Commun. Assoc., Sept. 2005, pp. 1833–1836.
[26] J. Hancock, L. Curry, S. Goorha, and M. Woodworth, “On lying and being lied to: A linguistic
analysis of deception in computer­mediated communication,” Discourse Processes, vol. 45,
pp. 1–23, Jan. 2008.
[27] V. Pérez­Rosas, C. Bologa, M. Burzo, and R. Mihalcea, “Deception detection within and
across cultures,” in Text Mining, Springer, Jan. 2008, pp. 157–175.
[28] E. Fitzpatrick, J. Bachenko, and T. Fornaciari, “Automatic detection of verbal deception,”
Synthesis Lectures on Human Language Technologies, vol. 8, no. 3, pp. 1–119, 2015.
[29] J. Allwood, L. Cerrato, K. Jokinen, C. Navarretta, and P. Paggio, “The mumin coding scheme
for the annotation of feedback, turn management and sequencing phenomena,” Lang. Resour.
and Eval., vol. 41, pp. 273–287, Dec. 2007.
[30] J. Li, Y. Wang, C. Wang, Y. Tai, J. Qian, J. Yang, C. Wang, J. Li, and F. Huang, “DSFD: Dual
shot face detector,” in 2019 Proc. IEEE Conf. Comput. Vision and Pattern Recognit., Jun.
2019, pp. 5060–5069.
[31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large­scale image
recognition,” in Proc. Int. Conf. Learn. Representat., 2015, pp. 1–14.
[32] E. Ramdinmawii, A. Mohanta, and V. K. Mittal, “Emotion recognition from speech signal,”
vol. 5, pp. 1599–1605, Apr. 2018.
[33] C.­Y. Chi, W.­C. Li, and C.­H. Lin, Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications. Boca Raton, FL, USA: CRC Press, 2017.
[34] V. Pérez­Rosas, M. Abouelenien, R. Mihalcea, Y. Xiao, C. Linton, and M. Burzo, “Verbal and
nonverbal clues for real­life deception detection,” in 2020 Proc. Conf. Empirical Methods
Natural Lang. Process., Sept. 2015, pp. 2336–2346.
[35] Z. Wu, B. Singh, L. S. Davis, and V. Subrahmanian, “Deception detection in videos,” in Proc.
AAAI Conf. Artif. Intell., Apr. 2018, pp. 1695–1702.