非結核分枝桿菌 (Non-tuberculous mycobacteria, NTM) 是一種會在人體引起肺部相關病症的分枝桿菌。在正常情況下，當非結核分枝桿菌入侵人體時，免疫系統有能力和其抗衡使其無法對人體造成損害，此平衡狀態稱為「移生(Colonization)」。當免疫系統因故減弱時，此時入侵的非結核分枝桿菌便會在人體內產生侵犯反應，稱之為「感染(Infection/Disease)」。實務上在辨別非結核分枝桿菌的移生或感染狀態時，會由多名醫師針對患者的肺部電腦斷層影像與痰液樣本的培養分析結果進行討論，取得共識後再做出最終的結果判定，因此通常耗時較長且耗費醫療資源。因此，若能找到一個有效且高效的針對非結核分枝桿菌的移生或感染狀態的辨別方法，對於醫療人員而言將會減輕許多負擔。在這篇論文中，針對猶如非結核分枝桿菌的移生或感染狀態此種非二元標籤的二元分類任務，我們提出了一個基於三維卷積神經網路(3D convolutional neural network, 3D CNN)與多層感知器(multilayer perceptron, MLP)的混合式深度學習模型，以及一種新的資料標籤化方法—權重資料標籤化(Weighted Labelling)，來根據患者的肺部電腦斷層影像與診療資料預測其體內非結核分枝桿菌的移生或感染狀態。實驗結果顯示採用我們提出的混和模型架構與權重資料標籤化的模型表現優於單純採用卷積神經網路或多層感知器的模型，亦優於未採用權重資料標籤化的模型。此外，我們亦設計了一個人機迴圈(Human-in-the-Loop, HITL)流程，透過醫生與模型的預測結果和關注熱度圖多個回合的互動與回饋，幫助醫生針對未取得共識的病例進行重新評估。根據我們的實驗結果，採用人機迴圈可以些微增進模型預測的準確率。在醫生重新評估的結果上，亦能看到前一個回合未能取得共識的病例，在採用人機迴圈後的共識率有所提升。針對參與人機迴圈流程的醫生設計的問卷回饋結果亦顯示出多數醫生認為在人機迴圈流程中，模型的預測結果是合理的，且對於未取得共識病例的重新評估是有幫助的。

Non-tuberculous mycobacteria (NTM) are a type of mycobacteria that cause pulmonary disease in humans. Normally, the human immune system is able to maintain a balance with the invading NTM bacteria, which is called Colonization. When the immune system becomes weak, NTM will cause Disease and bring about further injury to the human body. Distinguishing between Colonization and Disease requires several doctors to agree on a final diagnosis by analysing computed tomography (CT) scans and bacterial samples taken from patient's sputum, which is a time-consuming process. It is therefore helpful for doctors to find an effective and efficient way to identify NTM. In this work, we propose a mixture model of 3D convolutional neural network (CNN) and multilayer perceptron (MLP) and a labelling method called Weighted Labelling for a binary classification task with non-binary training data (such as NTM), which can automatically identify a suspected NTM patient using CT scans and clinical data. The mixture model with weighted labelling comprehensively outperforms models based on CNN or MLP alone, or without our proposed labelling method. We also develop a Human-in-the-Loop (HITL) process to help doctors re-evaluate cases without consensus by interacting with our model through the predictions and attention heat maps it generates for several rounds. Our experiments show that applying Human-in-the-Loop (HITL) process can slightly enhance the performance of the mixture model we proposed. The analysis of re-evaluation for each doctor also shows that doctors are more easier to have consensus when re-evaluating cases which are unconsensus in previous round after referencing predictions and attention heat maps from our model. Besides, the feedback of questionnaire show that the outputs of our model is reasonable and useful for re-evaluation on unconsensus cases for doctors during the Human-in-the-Loop (HITL) process.

Abstract (Chinese) I
Abstract II
Acknowledgements (Chinese) III
Contents IV
List of Figures VII
List of Tables IX
1 Introduction 1
2 Related Works 3
2.1 3D Convolutional Neural Network and Uniformizing Methods for 3D Computed Tomography Scans 3
2.2 Attention Mechanism on Convolutional Neural Network 5
2.3 Human-in-the-Loop Machine Learning 9
3 Dataset 12
3.1 Non-tuberculous Mycobacteria 12
3.2 Data 14
4 Methodology 16
4.1 Data Pre-processing 16
4.2 Lung Segmentation As Attention Map 20
4.3 Modeling 21
4.3.1 3D Convolutional Neural Network 21
4.3.2 Multilayer Perceptron 23
4.3.3 Mixture Model of 3D Convolutional Neural Network and
Multilayer Perceptron 24
4.4 Labelling 25
4.5 Human-in-the-Loop Design 26
4.6 Model Evaluation 29
4.7 Model Interpretation 29
4.7.1 Gradient-weighted Class Activation Mapping 29
4.7.2 Integrated Gradients 30
4.7.3 Mixture of Gradient-weighted Class Activation Mapping and
Integrated Gradients 31
5 Results 32
5.1 Dataset Hierarchy 32
5.2 Results of Different Architectures 35
5.2.1 Overwiew 35
5.2.2 Experiment Settings 35
5.2.3 Results 35
5.3 Model Interpretation of Different Architectures 36
5.4 Results of Different Labelling Methods 39
5.4.1 Overwiew 39
5.4.2 Experiment Settings 39
5.4.3 Results 39
5.5 Results of Human-in-the-Loop Design 40
5.5.1 Overwiew 40
5.5.2 Experiment Settings 41
5.5.3 Results 41
6 Discussion 43
7 Conclusion 45
Bibliography 46

[1] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access, 2018.
[2] Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. Power to the people: The role of humans in interactive machine learning. AI Magazine, 35(4), 2014.
[3] Reza Azad, Maryam Asadi-Aghbolaghi, Mahmood Fathy, and Sergio Escalera. Bi-directional convlstm u-net with densley connected convolution. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2019.
[4] AS Becker, C Bl¨uthgen, C Sekaggya-Wiltshire, B Castelnuovo, A Kambugu, J Fehr, T Frauenfelder, et al. Detection of tuberculosis patterns in digital photographs of chest x-ray images using deep learning: feasibility study. The International Journal of Tuberculosis and Lung Disease (IJTLD), 2018.
[5] Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv, 2015.
[6] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
[7] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object and stuff segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015.
[8] Arpad Dani. Colonization and infection. Central European journal of urology, 67(1), 2014.
[9] Carl De Boor. Bicubic spline interpolation. Journal of Mathematics and Physics, 41(1-4), 1962.
[10] Sagi Eppel. Hierarchical semantic segmentation using modular convolutional neural networks. arXiv, 2017.
[11] Sagi Eppel. Classifying a specific image region using convolutional nets with an roi mask as input. arXiv, 2018.
[12] Monika Grewal, Muktabh Mayank Srivastava, Pulkit Kumar, and Srikrishna Varadarajan. Radnet: Radiologist level accuracy using deep learning for hemorrhage detection in ct scans. IEEE International Symposium on Biomedical Imaging (ISBI), 2018.
[13] David E Griffith, Timothy Aksamit, Barbara A Brown-Elliott, Antonino Catanzaro, Charles Daley, Fred Gordin, Steven M Holland, Robert Horsburgh, Gwen Huitt, Michael F Iademarco, Michael Iseman, Kenneth Olivier, Stephen Ruoss, C. Fordham von Reyn, Richard J. Wallace Jr., and Kevin Winthrop. An official ATS/IDSA statement: diagnosis, treatment, and prevention of nontuberculous mycobacterial diseases., volume 175. American Thoracic Society, 2007.
[14] Charles S Haworth, John Banks, Toby Capstick, Andrew J Fisher, Thomas Gorsuch, Ian F Laurenson, Andrew Leitch, Michael R Loebinger, Heather J Milburn, Mark Nightingale, Peter Ormerod, Delane Shingadia, David Smith, Nuala Whitehead, Robert Wilson, and R Andres Floto. British thoracic society guidelines for the management of non-tuberculous mycobacterial pulmonary disease (ntm-pd). Thorax, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
[16] Lars Holmberg, Paul Davidsson, and Per Linde. A feature space focus in machine teaching. IEEE international conference on pervasive computing and communications workshops (PerCom Workshops), 2020.
[17] Kai-Lung Hua, Che-Hao Hsu, Shintami Chusnul Hidayati,Wen-Huang Cheng, and Yu-Jen Chen. Computer-aided classification of lung nodules on computed tomography images via deep learning technique. OncoTargets and therapy, 2015.
[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
[19] Xiaojie Huang, Junjie Shan, and Vivek Vaidya. Lung nodule detection in ct using 3d convolutional neural networks. IEEE International Symposium on Biomedical Imaging (ISBI), 2017.
[20] Paras Lakhani and Baskaran Sundaram. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology, 2017.
[21] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. IEEE/RSJ international conference on intelligent robots and systems (IROS), 2015.
[22] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. IEEE international conference on 3D vision (3DV), 2016.
[23] Robert Munro Monarch. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster, 2021.
[24] Eduardo Mosqueira-Rey, Elena Hern´andez-Pereira, David Alonso-R´ıos, Jos´e Bobes-Bascar´an, and ´Angel Fern´andez-Leal. Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review, 56(4), 2023.
[25] Steve Nowee. Directing attention of convolutional neural networks. 2017.
[26] C. S. PEIRCE. The numerical measure of the success of predictions. Science, NS-4(93), 1884.
[27] Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction, 2020.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems (NIPS), 2015.
[29] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision (ICCV), 2017.
[30] Burr Settles. Active learning literature survey. CS Technical Reports, University of Wisconsin-Madison Department of Computer Sciences, 2009.
[31] Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems (NIPS), 2015.
[32] Patrice Y. Simard, Saleema Amershi, David M. Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Verwey. Johan, Mo Wang, and John Wernsing. Machine teaching: A new paradigm for building machine learning systems. arXiv, 2017.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[34] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. International conference on machine learning (ICML), 2017.
[35] W. J Youden. Index for rating diagnostic tests. Cancer, 3(1), 1950.
[36] Liang Zou, Jiannan Zheng, Chunyan Miao, Martin J Mckeown, and Z Jane Wang. 3d cnn based automatic diagnosis of attention deficit hyperactivity disorder using functional and structural mri. IEEE Access, 2017.
[37] Hasib Zunair, Aimon Rahman, and Nabeel Mohammed. Estimating severity from ct scans of tuberculosis patients using 3d convolutional nets and slice selection. CLEF (Working Notes), 2019.
[38] Hasib Zunair, Aimon Rahman, Nabeel Mohammed, and Joseph Paul Cohen. Uniformizing techniques to process ct scans with 3d cnns for tuberculosis prediction. Predictive Intelligence in Medicine: Third International Workshop, MICCAI, 2020.