隨著科技的進步，人類資料和訊號的收集也越來越容易。其中，行為訊號處理變成一個很熱門的研究領域，並且在各方面都有良好的應用，而這些應用都指向一件事：「人類的訊號是可以被量化的」。在腦神經科學的領域，fMRI扮演著非常重要的角色。如何正確及合理的量化fMRI訊號成為一件很重要的議題。在此論文中，我們要用迴旋積自編碼來去建立一個fMRI的特徵擷取器。這個特徵擷取器是一個資料導向的特徵擷取器，也就是說它不帶有任何人類的偏見在裡面。並且我們會些用靜息態的fMRI大腦資料來對這個迴旋積自編碼做預訓練，並且證明這個模型已經學到了一些關於靜息態fMRI的一些性質。預訓練的目的有二，在深度學習方面，預訓練能讓我們的模型更強健，而且能給我們的模型一個很好的基礎權重值。在腦神經科學方面，我們能用解碼器來解碼並且分析模型確實有學會預訓練資料的性質。最後在情緒認知分類的應用中，我們在被IEMOCAP所刺激的fMRI資料分三種類別時，情緒的激動程度可以達到91.21%而在情緒的正負向則是93.42%(分別進步了13.16%和21.25%)。而在被NNIME刺激的fMRI大腦分兩類時，可以在情緒的激動程度和正負向分別到達72.53% 及74.44%(分別進步12.88% 和 20.58%)。我們的分析更進一步的指出我們的模型可以有轉移到不同性質的資料庫上面。

With the improvement of technology, the signal of human could be collected more easily, behavior signal processing (BSP) become a popular field for researchers in several applications, and they point out “Signal of human is measurable”. For Functional magnetic resonance imaging (fMRI), how to extract feature reasonably is an important issue. In this paper, we build a feature extractor for fMRI by convolutional autoencoder (CAE). This feature extractor is data-driven, without artificial bias. This CAE is pertained by resting states fMRI data, and we prove this CAE model learn the property of pretrain data. There are two purposes of pretrain, for deep learning domain, pretraining will make model more robust, and give better initial weights. For neuroscience domain, we find the network in brain via the analyze of decoder. The application of this model is used for two emotion cognitive classification tasks. In the IEMOCAP, the three class UAR reach 91.21% on arousal and 93.42% on valence. (13.16% and 21.25% improvement compared to the ICA and PCA). In the NNIME, the two class UAR reach 72.53% and 74.44% for arousal and valence. (12.88% and 20.58% improvement compared to the ICA and PCA). Our analysis further shows the model have ability of transfer to different property datasets.

Acknowledge i
中文摘要 ii
ABSTRACT iii
Chapter 1 Introduction 1
Chapter 2 fMRI Introduction & Datasets 4
2.1 fMRI Introduction 4
2.2 fMRI preprossing 6
2.3 Datasets and Vocal Emotion Stimuli Design 8
2.3.1 FCP1000 - Beijing_Zang part 8
2.3.2 NNIME 8
2.3.3 IEMOCAP-VAL and IEMOCAP-ACT 10
Chapter 3 Research Methodology 11
3.1 ICA 11
3.2 PCA 12
3.3 Convolutional Autoencoder 13
3.4 Data augmentation 19
3.5 Support Vector Machine 20
Chapter 4 Experimental Setup and Results 22
4.1 Experiment I 24
4.2 Experiment II 29
4.3 Experiment III 34
Chapter 5 Conclusion & Future Work 38
Reference 40

[1] Picard, R. W. (1995). Affective computing.
[2] Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D'Errico, F., & Schroeder, M. (2012). Bridging the gap between social animal and unsocial machine: A survey of social signal processing. IEEE Transactions on Affective Computing, 3(1), 69-87.
[3] ]Narayanan, S., & Georgiou, P. G. (2013). Behavioral signal processing: Deriving human behavioral informatics from speech and language. Proceedings of the IEEE, 101(5), 1203-1233.
[4] Hsiao, S. W., Sun, H. C., Hsieh, M. C., Tsai, M. H., Tsao, Y., & Lee, C. C. (2017). Toward Automating Oral Presentation Scoring during Principal Certification Program using Audio-video Low-level Behavior Profiles. IEEE Transactions on Affective Computing.
[5] Tsai, F. S., Hsu, Y. L., Chen, W. C., Weng, Y. M., Ng, C. J., & Lee, C. C. (2016). Toward Development and Evaluation of Pain Level-Rating Scale for Emergency Triage based on Vocal Characteristics and Facial Expressions. In INTERSPEECH (pp. 92-96).
[6] Lin, W. C., & Lee, C. C. (2018). Computational Analyses of Thin-sliced Behavior Segments in Session-level Affect Perception. IEEE Transactions on Affective Computing.
[7] Chen, C. P., Tseng, X. H., Gau, S. S. F., & Lee, C. C. (2017). Computing Multimodal Dyadic Behaviors during Spontaneous Diagnosis Interviews toward Automatic Categorization of Autism Spectrum Disorder. Age (Avg/Std), 14, 3-08.
[8] Chen, H. Y., Liao, Y. H., Jan, H. T., Kuo, L. W., & Lee, C. C. (2016, March). A Gaussian mixture regression approach toward modeling the affective dynamics between acoustically-derived vocal arousal score (VC-AS) and internal brain fMRI bold signal response. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on (pp. 5775-5779). IEEE.
[9] Zhang, L., Samaras, D., Tomasi, D., Volkow, N., & Goldstein, R. (2005). Machine learning for clinical diagnosis from functional magnetic resonance imaging (pp. 1211-1217). IEEE.
[10] Xie, S. Y., Guo, R., Li, N. F., Wang, G., & Zhao, H. T. (2009, June). Brain fMRI processing and classification based on combination of PCA and SVM. In Neural Networks, 2009. IJCNN 2009. International Joint Conference on (pp. 3384-3389). IEEE.
[11] Londei, A., D’Ausilio, A., Basso, D., Sestieri, C., Del Gratta, C., Romani, G. L., & Belardinelli, M. O. (2007). Brain network for passive word listening as evaluated with ICA and Granger causality. Brain Research Bulletin, 72(4-6), 284-292.
[12] Satterthwaite, T. D., Wolf, D. H., Loughead, J., Ruparel, K., Valdez, J. N., Siegel, S. J., ... & Gur, R. C. (2010). Association of enhanced limbic response to threat with decreased cortical facial recognition memory response in schizophrenia. American Journal of Psychiatry, 167(4), 418-426.
[13] Bangert, M., Peschel, T., Schlaug, G., Rotte, M., Drescher, D., Hinrichs, H., ... & Altenmüller, E. (2006). Shared networks for auditory and motor processing in professional pianists: evidence from fMRI conjunction. Neuroimage, 30(3), 917-926.
[14] Ochsner, K. N., Bunge, S. A., Gross, J. J., & Gabrieli, J. D. (2002). Rethinking feelings: an FMRI study of the cognitive regulation of emotion. Journal of cognitive neuroscience, 14(8), 1215-1229.
[15] Greene, J. D., Sommerville, R. B., Nystrom, L. E., Darley, J. M., & Cohen, J. D. (2001). An fMRI investigation of emotional engagement in moral judgment. Science, 293(5537), 2105-2108.
[16] Northoff, G., Heinzel, A., Bermpohl, F., Niese, R., Pfennig, A., Pascual‐Leone, A., & Schlaug, G. (2004). Reciprocal modulation and attenuation in the prefrontal cortex: an fMRI study on emotional–cognitive interaction. Human Brain Mapping, 21(3), 202-212.
[17] Raichle, M. E., MacLeod, A. M., Snyder, A. Z., Powers, W. J., Gusnard, D. A., & Shulman, G. L. (2001). A default mode of brain function. Proceedings of the National Academy of Sciences, 98(2), 676-682.
[18] Andrews-Hanna, J. R. (2012). The brain’s default network and its adaptive role in internal mentation. The Neuroscientist, 18(3), 251-270.
[19] Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., ... & Varoquaux, G. (2014). Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics, 8, 14.
[20] Meszlényi, R. J., Buza, K., & Vidnyánszky, Z. (2017). Resting state fMRI functional connectivity-based classification using a convolutional neural network architecture. Frontiers in neuroinformatics, 11, 61.
[21] Kamonsantiroj, S., Charoenvorakiat, P., & Pipanmaekaporn, L. (2016, July). Learning Representation for fMRI Data Analysis Using Autoencoder. In Advanced Applied Informatics (IIAI-AAI), 2016 5th IIAI International Congress on (pp. 560-565). IEEE.
[22] Huang, H., Hu, X., Zhao, Y., Makkie, M., Dong, Q., Zhao, S., ... & Liu, T. (2017). Modeling task fMRI data via deep convolutional autoencoder. IEEE transactions on medical imaging.
[23] Chen, P. H., Zhu, X., Zhang, H., Turek, J. S., Chen, J., Willke, T. L., ... & Ramadge, P. J. (2016). A convolutional autoencoder for multi-subject fMRI data aggregation. arXiv preprint arXiv:1608.04846.
[24] Yan, C., & Zang, Y. (2010). DPARSF: a MATLAB toolbox for" pipeline" data analysis of resting-state fMRI. Frontiers in systems neuroscience, 4, 13.
[25] Brett M, Hanke M, Cipollini B, Côté M, Markiewicz C, Gerhard S, Larson E, Lee G, Halchenko Y, Kastman E, cindeem, Morency F, moloney, Millman J, Rokem A, jaeilepp, Gramfort A, den Bosch JFv, Subramaniam K, Nichols N, embaker, bpinsard, chaselgrove, Oosterhof N, St-Jean S, Amirbekian B, Nimmo-Smith I, Ghosh S, Varoquaux G, Garyfallidis E (2016) nibabel: 2.1.0. Zenodo https://doi.org/10.5281/ ZENODO.60808
[26] Biswal, B. B., Mennes, M., Zuo, X. N., Gohel, S., Kelly, C., Smith, S. M., ... & Dogonowski, A. M. (2010). Toward discovery science of human brain function. Proceedings of the National Academy of Sciences, 107(10), 4734-4739.
[27] Chou, H. C., Lin, W. C., Chang, L. C., Li, C. C., Ma, H. P., & Lee, C. C. (2017, October). NNIME: The NTHU-NTUA Chinese interactive multimodal emotion corpus. In Affective Computing and Intelligent Interaction (ACII), 2017 Seventh International Conference on (pp. 292-298). IEEE.
[28] Busso, C., Bulut, M., Lee, C. C., Kazemzadeh, A., Mower, E., Kim, S., ... & Narayanan, S. S. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4), 335.
[29] Lee, T. W. (1998). Independent component analysis. In Independent component analysis (pp. 27-66). Springer, Boston, MA.
[30] Calhoun, V. D., Liu, J., & Adalı, T. (2009). A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data. Neuroimage, 45(1), S163-S172.
[31] Varoquaux, G., Sadaghiani, S., Pinel, P., Kleinschmidt, A., Poline, J. B., & Thirion, B. (2010). A group model for stable multi-subject ICA on fMRI datasets. Neuroimage, 51(1), 288-299.
[32] Xie, J., Douglas, P. K., Wu, Y. N., Brody, A. L., & Anderson, A. E. (2017). Decoding the encoding of functional brain networks: An fMRI classification comparison of non-negative matrix factorization (NMF), independent component analysis (ICA), and sparse coding algorithms. Journal of neuroscience methods, 282, 81-94.
[33] Jolliffe, I. (2011). Principal component analysis. In International encyclopedia of statistical science (pp. 1094-1096). Springer, Berlin, Heidelberg.
[34] Lama, R. K., Gwak, J., Park, J. S., & Lee, S. W. (2017). Diagnosis of Alzheimer’s disease based on structural MRI images using a regularized extreme learning machine and PCA features. Journal of healthcare engineering, 2017.
[35] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
[36] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).
[37] Zeiler, M. D., & Fergus, R. (2014, September). Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833). Springer, Cham.
[38] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
[39] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
[40] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
[41] Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., ... & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical image analysis, 42, 60-88.
[42] Baldi, P. (2012, June). Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupervised and transfer learning (pp. 37-49).
[43] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[44] Lin, Y. L., & Wei, G. (2005, August). Speech emotion recognition based on HMM and SVM. In Machine Learning and Cybernetics, 2005. Proceedings of 2005 International Conference on (Vol. 8, pp. 4898-4901). IEEE.
[45] Nasreen, A., Vinutha, H., & Shobha, G. (2018). Analysis of Video Content Through Object Search Using SVM Classifier. In Innovations in Electronics and Communication Engineering (pp. 325-333). Springer, Singapore.
[46] Kaucha, D. P., Prasad, P. W. C., Alsadoon, A., Elchouemi, A., & Sreedharan, S. (2017, September). Early detection of lung cancer using SVM classifier in biomedical image processing. In 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI) (pp. 3143-3148). IEEE.
[47] Lieberman, Matthew (2 September 2016). Social. Broadway Books. p. 19. ISBN 978-0-307-88910-2.
[48] Greicius, M. D., Krasnow, B., Reiss, A. L., & Menon, V. (2003). Functional connectivity in the resting brain: a network analysis of the default mode hypothesis. Proceedings of the National Academy of Sciences, 100(1), 253-258.
[49] Penny, W. D., Friston, K. J., Ashburner, J. T., Kiebel, S. J., & Nichols, T. E. (Eds.). (2011). Statistical parametric mapping: the analysis of functional brain images. Elsevier.