我們從心理學的角度提出了基於五官特徵的表情辨識方法，利用心理學家Paul Ekman的Action Units論點，藉由探索五官在作表情時的肌肉變化，將臉再分成五官區域來作辨識。本論文將從諸多相關研究中從介紹美國心理學家Paul Ekman開始，與六個基礎表情，以及特徵擷取或五官模型為主的表情辨識方法，而本論文是以實際測試的辨識系統作切入，探討研究每個步驟，流程大致有兩大部分：「前處理」與「辨識方法」。

在第一部分，由於實驗環境與實際測試環境的差異，像是燈光亮度的不同，或是每個人的臉型、膚色等先天條件的不同，會顯著影響辨識的準確度，因此我們提出前處理來解決此問題：首先將輸入的影像做人臉偵測，根據偵測結果找尋五官點，接著再根據眼睛所構成之水平軸線做旋轉校正，再依照人臉模型正確擷取出完整的人臉與做燈光校正，並縮放至相同大小以利後面的特徵擷取。

前處理後影像上人臉的差異化變小，這時便進入第二部分「辨識方法」。這裡我們主要採用賈波濾波器（Gabor filter bank）這個特徵擷取方法，搭配區塊擷取（ROI capture），並根據擷取後的高維度特徵向量，使用主成份分析（principal component analysis）加上線性識別分析（linear discriminant analysis）做降維，以增加辨識速度，再搭配支持向量機（support vector machine）作為我們的分類器。

在實驗分析上，我們提出的基於五官特徵的表情辨識方法在既有的JAFFE、TFEID和CK+資料庫的最佳辨識率（測量標準為leave-one-person-out），分別能達到86.1%、96.9%和89.0%，而在我們自己所蒐集的101SC資料庫，相對較難但也更符合實際狀況的辨識率能達到62.1%。本論文所提出的方法亦參加了2013年第八屆由田機器視覺獎，在10個參賽隊伍中獲得第二名的成績。

We propose an expression recognition method based on facial features from the psychological perspective. According to the American psychologist Paul Ekman’s work on action units, we divide a face into different facial feature regions for expression recognition via the movements of individual facial muscles during slight different instant changes in facial expression. This thesis starts from introducing Paul Ekman’s work, 6 basic emotions, and existing methods based on feature extraction or facial models. Our system have two main parts: preprocessing and recognition method.

The difference in training and test environments, such as illumination, or face size and skin color of different subjects under testing, is usually the major influencing factor in recognition accuracy. It is therefore we propose a preprocessing step in our first part of the system: we first perform face detection and facial feature detection to locate facial features. We then perform a rotation calibration based on the horizontal line obtained by connecting both eyes. The complete face region can be extracted by using facial models. Lastly, the face region is calibrated for illumination and resized to same resolution for dimensionality of feature vector.

After preprocessing, we can reduce the difference among images. Second part of our proposed system is the recognition method. Here we use Gabor filter banks with ROI capture to obtain the feature vector and principal component analysis (PCA) and linear discriminant analysis (LDA) for dimensionality reduction to reduce the computation time. Finally, a support vector machine (SVM) is adopted as our classifier.

The experimental result shows that the proposed method can archive 86.1%, 96.9%, and 89.0% accuracy on three existing datasets JAFFE, TFEID, and CK+ respectively (based on leave-one-person-out evaluation). We also tested the performance on the 101SC dataset that were collected and prepared by ourselves. This dataset is relatively difficult in recognition but closer to the scenario in reality. The proposed method is able to achieve 62.1% accuracy on it. We also use this method to participate the 8th UTMVP (Utechzone Machine Vision Prize) competition, and we were ranked the second place out of 10 teams.

摘要 I
Abstract II
謝誌 IV
目錄 VI
表目次 X
圖目次 XI
第一章 緒論 1
1.1 研究主題 1
1.1.1 研究簡介 1
1.1.2 競賽簡介 2
1.2 相關研究簡介 3
1.2.1 Paul Ekman 3
1.2.2 六大表情說明 4
1.2.3 Action Units 4
1.2.4 基於賈波濾波器與降維的人臉表情辨識 7
1.2.5 使用自適應增強作特徵選擇的人臉表情辨識 8
1.2.6 雙子空間非負矩陣分解應用於人臉表情辨識 9
1.2.7 主動形狀模型 10
1.2.8 基於外觀模型的人臉表情辨識與統整 11
1.2.9 Haar-like特徵 12
1.2.10 LBP 13
1.3 章節概要 14
第二章 研究方法 15
2.1 系統架構 15
2.2 影像前處理 15
2.2.1 灰階化 16
2.2.2 人臉偵測 17
2.2.3 五官點偵測 19
2.2.4 人臉校正 20
2.2.5 影像縮放 20
2.2.6 燈光校正 21
2.3 影像特徵擷取 22
2.3.1 Gabor Filter Bank 23
2.4 區塊擷取 24
2.4.1 五官特徵在表情表現上的討論 27
2.5 降維 29
2.5.1 PCA 30
2.5.2 LDA 31
2.5.3 降維總結討論 32
2.6 分類器 33
2.6.1 支持向量機 33
第三章 研究結果與分析 36
3.1 表情辨識資料庫說明 36
3.1.1 JAFFE 37
3.1.2 JAFFE資料討論 38
3.1.3 TFEID 38
3.1.4 CK+ 40
3.1.5 101SC 41
3.2 實驗設定說明 42
3.2.1 參數設定 43
3.2.2 效能評估方法 43
3.3 前處理實驗結果 44
3.3.1 人臉校正之實驗設定 44
3.3.2 人臉校正組合辨識率分析 44
3.3.3 人臉校正為何需要之說明 45
3.4 表情辨識結果 46
3.4.1 簡述比較之現有方法 46
3.4.2 各辨識方法組合辨識率分析 47
3.4.3 ROI Capture實驗分析討論 48
3.4.4 五官特徵的表情辨識實驗分析 49
3.4.5 五官特徵與全臉的表情辨識討論 51
3.5 錯誤分析 52
第四章 結論與未來研究方向 54
4.1 結論 54
4.2 未來研究方向 55
參考文獻 57

[1] 由田機器視覺獎http://www.utmvp.com/
[2] http://en.wikipedia.org/wiki/Paul_Ekman
[3] P. Ekman and W. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978.
[4] Paul Ekman, Wallace V. Friesen, and Joseph C. Hager. Facial Action Coding System: The Manual on CD ROM. A Human Face, Salt Lake City, 2002.
[5] Hamm, J.; Kohler, C. G.; Gur, R. C.; Verma, R. (2011). "Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders". Journal of Neuroscience Methods 200 (2): 237–256.
[6] Zhang, Lei, Yan Tong, and Qiang Ji. "Active image labeling and its application to facial action labeling." Computer Vision–ECCV 2008. Springer Berlin Heidelberg, 2008. 706-719.
[7] Deng, Hong-Bo, et al. "A new facial expression recognition method based on local gabor filter bank and pca plus lda." International Journal of Information Technology 11.11 (2005): 86-96.
[8] Shih, Frank Y., and Chao-Fa Chuang. "Automatic extraction of head and face boundaries and facial features." Information Sciences 158 (2004): 117-130.
[9] Silapachote, Piyanuch, Deepak R. Karuppiah, and Allen R. Hanson. Feature selection using AdaBoost for face expression recognition. MASSACHUSETTS UNIV AMHERST DEPT OF COMPUTER SCIENCE, 2005.
[10] Tu, Yi-Han, and Chiou-Ting Hsu. "Dual subspace nonnegative matrix factorization for person-invariant facial expression recognition." Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 2012.
[11] Blake, Andrew, and Michael Isard. Active shape models. Springer London, 1998.
[12] Abboud, Bouchra, Franck Davoine, and Mo Dang. "Facial expression recognition and synthesis based on an appearance model." Signal Processing: Image Communication 19.8 (2004): 723-740.
[13] vision.CascadeObjectDetector System object http://www.mathworks.com/help/vision/ref/vision.cascadeobjectdetectorclass.html
[14] Lienhart R., Kuranov A., and V. Pisarevsky "Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection.", Proceedings of the 25th DAGM Symposium on Pattern Recognition. Magdeburg, Germany, 2003.
[15] Ojala Timo, Pietikäinen Matti, and Mäenpää Topi, “Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns” . In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002. Volume 24, Issue 7, pp. 971-987.b
[16] Viola, Paul, and Michael Jones. "Robust real-time object detection." International Journal of Computer Vision 4 (2001).
[17] Viola, Paul, and Michael Jones. "Rapid object detection using a boosted cascade of simple features." Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. IEEE, 2001.
[18] Felzenszwalb, Pedro F., and Daniel P. Huttenlocher. "Pictorial structures for object recognition." International Journal of Computer Vision 61.1 (2005): 55-79.
[19] ``Hello! My name is... Buffy'' -- Automatic Naming of Characters in TV Video http://www.robots.ox.ac.uk/~vgg/publications/html/everingham06a-abstract.html
[20] Chang, Chih-Chung, and Chih-Jen Lin. "LIBSVM: a library for support vector machines." ACM Transactions on Intelligent Systems and Technology (TIST) 2.3 (2011): 27.
[21] Freund, Yoav, and Robert E. Schapire. "A desicion-theoretic generalization of on-line learning and an application to boosting." Computational learning theory. Springer Berlin Heidelberg, 1995.
[22] Freund, Yoav, and Robert E. Schapire. "Experiments with a new boosting algorithm." ICML. Vol. 96. 1996.
[23] Lyons, Michael, et al. "Coding facial expressions with gabor wavelets." Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on. IEEE, 1998.
[24] TFEID (Taiwanese Facial Expression Image Database) http://bml.ym.edu.tw/tfeid/
[25] Lucey, Patrick, et al. "The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression." Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, 2010.
[26] Dantone, Matthias, et al. "Real-time facial feature detection using conditional regression forests." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.
[27] Zhao, Quan-You, et al. "Facial expression recognition based on fusion of Gabor and LBP features." Wavelet Analysis and Pattern Recognition, 2008. ICWAPR'08. International Conference on. Vol. 1. IEEE, 2008.
[28] Lee, Hsi-Chieh, Chia-Ying Wu, and Tzu-Miao Lin. "Facial Expression Recognition Using Image Processing Techniques and Neural Networks." Advances in Intelligent Systems and Applications-Volume 2. Springer Berlin Heidelberg, 2013. 259-267.
[29] Gu, W. F., Y. V. Venkatesh, and C. Xiang. "Composite orthonormal basis for person-independent facial expression recognition." Industrial Engineering and Engineering Management (IEEM), 2010 IEEE International Conference on. IEEE, 2010.