在論文中，我們提出了一個基於生理訊號的情緒辨識系統，使人類與機器間 的互動關係能有更進一步的發展。我們收集了一組全新的情緒資料庫─台藝大資料庫（NTUA database）來研究與驗證此系統，並使用演出的方式來自發性地引導出受測者的情緒。資料庫中紀錄了受測者的影像資料、音訊資料及生理訊號（心電訊號及呼吸訊號），論文中，我們專注於生理訊號的部分，研究並使之成為可信賴的情緒辨別方法。
為了有效的辨識受測者的情緒，我們分析所記錄的情緒標示結果並利用這些結果建出一個二維的情緒效價與喚起模型（valence-arousal model），再者除了使用心律變異度分析（heart rate variability）中時域分析及頻域分析來擷取生理訊號的特徵外，我們更使用了混和高斯模型（Gaussian mixture model）和費雪向量（Fisher vector）來得到一組全新的特徵。在音訊或視訊處理上面，使用混和高斯模型和Fisher vector來萃取特徵是十分常見的，但在生理訊號的領域上卻很少使用，然而，此方法可以反映出一連串的生理反應，並增加辨識準確度。
我們選擇支持向量機當作分類器來區分不同等級的情緒激發程度和情緒正負面程度，搭配最小冗餘最大相關特徵選擇來選取較適合的特徵，並分析每個實驗步驟的辨識結果來驗證這些過程是合理的，並比較從心律變異分析和費雪向量擷取出兩種不同的特徵之情緒辨識的準確度，分類出來的結果準確度為激發程度61.1%，正負面程度為54.6%。此外，我們也融合了生理訊號與音訊這兩種訊號的特徵來做分類，最終的分類結果為情緒效價64.24%，情緒喚起為63.31%。

To improve the mutual relationship between humans and machines, an emotion recognition system based on physiological signals is proposed. To study and verify this system, we collect a new emotional database, the National Taiwan University of Art database (NTUA database), and use performing induction method to spontaneously elicit subjects' emotional states. The database records subjects' performance video, audio and physiological signals (electrocardiogram and respiration). In this work, we focus on the physiological signals, and try to make it a reliable channel of emotion recognition.
To recognize the subjects' emotion efficiently, we analyze the emotional labels and make use of them to build a two-dimensional valence-arousal model. In addition, not only the prevailing functional features from heart rate variability (HRV) in time domain and frequency domain, but also the features encoded by the Gaussian mixture mode (GMM) and Fisher vector are introduced to obtain new features. The latter is a common method in audio and video signal processing, but it is rarely conducted in physiological area. However, this method can truly reflect the series of physiological reaction, and improve the recognition accuracy.
The support vector machine is proposed as the classifier to classify different levels of valence and arousal, and minimum redundancy maximum relevance (mRMR) is used as the strategy of feature selection to pick up the useful features. We analyzed the recognition results from each experiment step to prove that each process was rational, and compared the recognition accuracy between the features from HRV and the features from Fisher vector. The final accuracy of classification is 54.6% for valence and 61.1% for arousal. We also conduct the feature-level fusion of physiological and audio signals, and the recognition accuracy increases to 64.24% for valence and 63.31% for arousal.

1. Introduction ......P1
2. Overview of Emotion Recognition Based on Physiological Signals ......P5
3. Experimental Setting ......P15
4. Proposed Methodology ......P27
5. Experimental Results and Discussion ......P51
6. Conclusions and Future Works ......P65

[1] C. H. Wu and W. B. Liang, “Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels,” IEEE Transactionson Affective Computing.
[2] K. Wang, N. An, B. N. Li, Y. Zhang, and L. Li, “Speech Emotion Recognition Using Fourier Parameters,” IEEE Transactions on Affective Computing, vol. 6, no. 1, pp. 69–75, Jan 2015.
[3] C. H. Wu, W. L. Wei, J. C. Lin, and W. Y. Lee, “Speaking Effect Removal on Emotion Recognition From Facial Expressions Based on Eigenface Conversion,” IEEE Transactions on Multimedia, vol. 15, no. 8, pp. 1732–1744, Dec 2013.
[4] F. Abdat, C. Maaoui, and A. Pruski, “Human-Computer Interaction Using Emotion Recognition from Facial Expression,” in Computer Modeling and Simulation (EMS), 2011 Fifth UKSim European Symposium on, Nov 2011, pp. 196–201.
[5] W. James, The Principles of Psychology. Read Books Ltd, 2013.
[6] C. J. de Luca, R. S. LeFever, M. P. McCue, and A. P. Xenakis, “A circumplex
model of affect,” J Physiol, vol. 39, pp. 113–128, 1982. [Online]. Available:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1224770/.
[7] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cogni-tive development, and psychopathology,” Develop. Psychopathol., vol. 17, no. 03, pp. 715–734, 2005.
[8] P.Ekman and W.V.Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.
[9] J. Song, Z. Chi, J. Liu, and H. Fu, “Extraction of face image edges with application to expression analysis,” in Control, Automation, Robotics and Vision Conference, 2004. ICARCV 2004 8th, vol. 2, Dec 2004, pp. 804–809 Vol. 2.
[10] J. F. Cohn, A. J. Zlochower, J. J. Lien, and T. Kanade, “Feature-point tracking by optical flow discriminates subtle differences in facial expression,” in Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, Apr 1998, pp. 396–401.
[11] I. Buciu, C. kotropoulos, and I. Pitas, “ICA and Gabor representation for facial expression recognition,” in Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, vol. 2, Sept 2003, pp. II–855–8 vol.3.
[12] B. Schuller, G. Rigoll, and M. Lang, “Speech Emotion Recognition Combining Acoustic Features and Linguistic Information in a Hybrid Support Vector Machine-Belief Network Architecture,” in Acoustics, Speech, and Signal Processing, 2004. Proceedings. (ICASSP ’04). IEEE International Conference on, vol. 1, May 2004, pp. I–577–80 vol.1.
[13] D.-N. Jiang and L.-H. Cai, “Speech Emotion Classification with the Combination of Statistic Features and Temporal Features,” in Multimedia and Expo, 2004. ICME ’04. 2004 IEEE International Conference on, vol. 3, June 2004, pp. 1967–1970 Vol.3.
[14] B. Schuller, G. Rigoll, and M. Lang, “Hidden Markov model-based speech emotion recognition,” in Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP’03). 2003 IEEE International Conference on, vol. 2, April 2003, pp. II–1–4 vol.2.
[15] X. H. Le, G. Quenot, and E. Castelli, “Recognizing Emotions for the Audio-Visual Document Indexing,” in Computers and Communications, 2004. Proceedings. ISCC 2004. Ninth International Symposium on, vol. 2, June 2004, pp. 580–584 Vol.2.
[16] J. Cai, G. Liu, and M. Hao, “The Research on Emotion Recognition from ECG Signal,” in Information Technology and Computer Science, 2009. ITCS 2009. International Conference on, vol. 1, July 2009, pp. 497–500.
[17] M. Nardelli, G. Valenza, A. Greco, A. Lanata, and E. P. Scilingo, “Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability,” IEEE Transactions on Affective Computing, vol. 6, no. 4, pp. 385–394, Oct 2015.
[18] H. W. Guo, Y. S. Huang, J. C. Chien, and J. S. Shieh, “Short-term analysis of heart rate variability for emotion recognition via a wearable ecg device,” in 2015 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), Nov 2015, pp. 262–265.
[19] M. M. Bradley and P. J. Lang, “The International Affective Digitized Aounds (; IADS-2): Affective ratings of sounds and instruction man-ual,” Univ. of Florida, Gainesville, FL, USA, Tech. Rep. B-3, 2007.
[20] J. Kim and E. Andr, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067–2083, Dec 2008.
[21] G. Valenza, A. Lanata, and E. P. Scilingo, “The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition,” IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 237–249, April 2012.
[22] P. J. Lang, M. M. Bradley, and B. N. . Cuthbert, “International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual (Rep. No. A-8),” ResearchGate, Jan. 2008. [Online]. Available: https://www.researchgate.net/publication/256309451 International Affective Picture System IAPS Affective Ratings of Pictures and Instruction Manual Rep No A-8
[23] K. H. Kim, S. W. Bang, and S. R. Kim, “Emotion Recognition System Using short-term monitoring of Physiological Signals,” Medical and Biological Engineering and Computing, vol. 42, no. 3, pp. 419–427. [Online]. Available: http://link.springer.com/article/10.1007/BF02344719
[24] C. D. Katsis, N. Katertsidis, G. Ganiatsas, and D. I. Fotiadis, “Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach,” IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 38, no. 3, pp. 502–512, May 2008.
[25] S. Anders, F. Eippert, N. Weiskopf, and R. Veit, “The Human Amygdala is Sensitive to the Valence of Pictures and Sounds Irrespective of Arousal: An fMRI study,” Social Cognitive and Affective Neuroscience, vol. 3, no. 3, pp. 233–243, Sep. 2008.
[26] M. P. Tarvainen, P. O. Ranta-aho, and P. A. Karjalainen, “An Advanced Detrending Method with Application to HRV Analysis,” IEEE Transactions on Biomedical Engineering, vol. 49, no. 2, pp. 172–175, Feb 2002.
[27] G. Valenza, L. Citi, and R. Barbieri, “Estimation of instantaneous complex dynamics through Lyapunov exponents: A study on heartbeat dynamics,” PloS One, vol. 9, no. 8, p. e105622, 2014.
[28] M. Casdagli, S. Eubank, J. D. Farmer, and J. Gibson, “State Space Reconstruction in the Presence of Noise,” Physica D: Nonlinear Phenomena, vol. 51, no. 1, pp. 52–98, Aug. 1991. [Online]. Available: http://www.sciencedirect.com/science/article/pii/016727899190222U
[29] S. Chen, Z. Gao, and S. Wang, “Emotion Recognition from Peripheral Physiological Signals enhanced by EEG,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2016, pp. 2827–2831.
[30] G. Yoo and S. Hong, “Emotion Evaluation Analysis and System Design of Biosignal,” in 2016 International Conference on Platform Technology and Service (PlatCon), Feb 2016, pp. 1–4.
[31] H. Schlosberg, “Three Dimensions of Emotion,” Psychological Rev., vol. 61, pp. 81–88, 1954.
[32] J.-W. Jhuang, “A Patch-sized Wearable ECG/Respiration Recording Platform with DSP Capability,” Master’s thesis, National Tsing Hua University, Sep. 2014.
[33] M. Mcllory and M. Cheithin, Clinical Cardiology, fifth ed. VLANGE Medical, 1990.
[34] C. Martin-Soelch, M. St¨ocklin, G. Dammann, K. Opwis, and E. Seifritz, “Anxiety trait modulates psychophysiological reactions, but not habituation processes related to affective auditory stimuli,” International journal of psychophysiology, vol. 61, no. 2, pp.87–97, 2006.
[35] A. Keil, M. M. Bradley, M. Junghfer, T. Russmann, W. Lowenthal, and P. J. Lang, “Cross-modal attention capture by affective stimuli: evidence from event-related potentials,” Cognitive, Affective & Behavioral Neuroscience, vol. 7, no. 1, pp. 18–24, Mar. 2007.
[36] J. Wtorek, A. Bujnowski, M. Lewandowska, J. Ruminski, and M. Kaczmarek, “Simultaneous monitoring of heart performance and respiration activity,” in 3rd International Conference on Human System Interaction, May 2010, pp. 661–665.
[37] C. Cai and P. d. B. Harrington, “Different Discrete Wavelet Transforms Applied to Denoising Analytical Data,” Journal of chemical information and computer sciences, vol. 38, no. 6, pp. 1161–1170, 1998.
[38] H.-G. Stark, “Continuous Wavelet Transform and Continuous Multiscale Analysis,” Journal of Mathematical Analysis and Applications, vol. 169, no. 1, pp. 179–196, Sep. 1992. [Online]. Available: http://www.sciencedirect.com/science/article/pii/0022247X9290110Y
[39] Q. Haibing, L. Xiongfei, and P. Chao, “Discrete wavelet soft threshold denoise processing for ecg signal,” in Intelligent Computation Technology and Automation (ICICTA), 2010 International Conference on, vol. 2, May 2010, pp. 126–129.
[40] K. C. Bilchick and R. D. Berger, “Heart rate variability,” Journal of cardiovascular electrophysiology, vol. 17, no. 6, pp. 691–694, 2006.
[41] M. Costa, A. L. Goldberger, and C.-K. Peng, “Multiscale entropy analysis of complex physiologic time series,” Physical review letters, vol. 89, no. 6, p. 068102, 2002.
[42] M. Brennan, M. Palaniswami, and P. Kamen, “Do existing measures of poincare plot geometry reflect nonlinear features of heart rate variability?” IEEE transactions on biomedical engineering, vol. 48, no. 11, pp. 1342–1347, 2001.
[43] F. Perronnin and C. Dance, “Fisher Kernels on Visual Vocabularies for Image Categorization,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007, pp. 1–8.
[44] “The MIT-BIH Normal Sinus Rhythm Database.” [Online]. Available: https: //physionet.org/physiobank/database/nsrdb/
[45] “Normal Sinus Rhythm RR Interval Database.” [Online]. Available: https://physionet.org/physiobank/database/nsr2db/
[46] B. McFee, R. Yamamoto, O. Nieto, A. Holovaty, J. F. Santos, D. Ellis, C. Raffel J. Moore, P. Viktorin, R. Bittner, E. Battenberg, D. Liang, D. Repetto, and M. McVicar, “librosa: 0.4.1,” Oct. 2015, dOI: 10.5281/zenodo.32193. [Online]
Available: http://zenodo.org/record/32193
[47] D. Reynolds, “Gaussian Mixture Models.”
[48] A. Vedaldi and B. Fulkerson, “VLFeat: An Open and Portable Library of Computer Vision Algorithms,” http://www.vlfeat.org/, 2008.
[49] M. Mandal and A. Mukhopadhyay, “An Improved Minimum Redundancy Maximum Relevance Approach for Feature Selection in Gene Expression Data,” Procedia Technology, vol. 10, pp. 20–27, 2013.
[50] G. Brown, A. Pocock, M.-J. Zhao, and M. Lujn, “Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection,” Journal of Machine Learning Research, vol. 13, p. 2766, Jan. 2012. [Online]. Available: http://jmlr.csail.mit.edu/papers/v13/brown12a.html
[51] T. Mitchell, “Computational learning theory,” in Machine Learning. McGraw-Hill Science/Engineering/Math, 1997, ch. 7, pp. 201–229.
[52] Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge University Press, 2000.
[53] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Transactions on Intelligent Systems and Technology, vol. 2, pp. 27:1–27:27, 2011, software available at http://www.csie.ntu.edu.tw/_cjlin/libsvm.