深度學習對於設備與資料有著非常高的要求[1]，其中資料更是模型好壞的核心。然而在實際問題上，我們要獲取標記好的資料是不容易的。在我們的論文中就是希望能解決這類型的問題，儘管手上的資料不一定非常的好，但還是能透過一些方法，得到一個結果不差的模型。
我們的實驗主要是在處理無人機影像與衛星影像的地物分析。這些資料數量不多，且不是這麼容易進行標記。我們透過GoogleAPI尋找類似我們實驗的資料集來進行訓練，然而我們收集的資料也不是這麼理想，有20%-30%的比例是標記錯誤的。
為了解決這個問題，我們藉由[2],[3],[4],[5],[6]..等類神經網路來設計我們的模型架構，並參考了[7], [8] ,[9]等非監督式模型。透過[10]可以先假定我們原始資料所訓練的模型是不差的，並結合blockwise的model來輔助我們修改標記，來解決標記錯誤的問題。為了取得更好的效果，我們將這個方法進行疊代，使得這筆GoogleAPI的資料集提升了3%~5%的準確度，並找到了我們想要的模型。
最後，當我們拿到一筆新的未標記資料時，可以在不使用其他資料集的前提下，只需進行一些簡單的標記，就可以利用我們的模型架構和演算法直接訓練出一個適合的模型，依此來解決難以取得有標記資料的問題。

Deep learning has very high requirements for equipment and materials [1], and the data is the core of the model. However, in practical matters, it is not easy for us to obtain the marked information. In our paper, we hope to solve this type of problem. Although the information we have is not good enough, we can still get a model with good results through some methods.
Our experiments are mainly in image segmentation of unmanned aerial vehicles and satellite imagery. The amount of this information is less and difficult to label, so we use GoogleAPI to find a data set similar to our experiment for training. However, the data we collected is not so ideal, and 20%-30% is labeled incorrectly.
In order to solve this problem, we design our model architecture by [2],[3],[4],[5],[6] and other neural networks, and reference to [7], [8], [9] and other unsupervised models. By [10], we can assume that the model we trained in the original data is not bad, then combine with the block-wise model to assist us in modifying the markup and solve the problem of markup errors. For better results, we use this method to iterate. Then we improved the 3%~5% accuracy in the GoogleAPI dataset and found the model we wanted.
Finally, when we get a new unlabeled dataset, we can use our model architecture and algorithms to directly train a suitable model without using other data sets. In order to solve the problem of difficult to obtain labeled data.

CONTENTS

口試委員會審定書 #
中文摘要 2
ABSTRACT 3
CONTENTS 4
LIST OF FIGURES 6
Chapter 1 Introduction 9
Chapter 2 Background 10
2.1 Introduction to Artificial Neural Networks 10
2.2 Backpropagation 12
2.3 Loss function and Overfitting 14
2.4 Convolution Neural Networks 16
Chapter 3 Neural Network Architecture 17
3.1 VGG 18
3.2 Residual Net 18
3.3 Fully convolution neural network 20
3.4 U-Net 20
3.5 Refine Net 21
Chapter 4 Algorithms and Implementation 22
4.1 Data and data processing 22
4.1.1 Motivation and Problem 22
4.2 Model structure 24
4.3 Processing data problems 25
4.3.1 Distribution of data 25
4.3.2 Fine-tuning 26
4.3.3 Self-transfer Learning 27
4.3.4 Transfer Learning with Modify Part of Data 28
4.3.5 Transfer Learning with Noise 30
4.3.6 Transfer Learning with Block-wise Prediction 33
Chapter 5 Experimental result 38
5.1 Results of methods with data selection 38
5.2 Results of methods without data selection 40
5.2.1 Transfer Learning with Noise on Mnist 41
5.2.2 Integrate the application of our method 44
Chapter 6 Experimental result 46
Reference 47

LIST OF FIGURES

Fig. 2.1 Comparing Perceptron and Logistic Regression 11
Fig. 2.2 An example of fully connect neural networks 11
Fig. 2.3 A simple example of a neural 12
Fig. 2.4 A picture that assists us to let us know backpropagation 13
Fig. 2.5 Backward Calculation Algorithm 13
Fig. 2.6 Rectified linear unit 14
Fig. 2.7 Dropout 15
Fig. 2.8 Convolution on Neural Networks 16
Fig. 2.9 Max pooling 17
Fig. 3.1 The experiment of very deep network model 19
Fig. 3.2 The principle of Residual Net 19
Fig. 3.3 The neural network architecture for FCNN 20
Fig. 3.4 The neural network architecture for U-net 21
Fig. 3.5 The neural network architecture for Refine net 21
Fig. 4.1 Pixel-wise labeling image in GoogleAPI 22
Fig. 4.2 The image with labeling error 23
Fig. 4.3 Our model structure 24
Fig. 4.4 Definition of our data set 25
Fig. 4.5 The flow chart of fine-tuning 26
Fig. 4.6 New label data of self-transfer learning 27
Fig. 4.7 The flow chart of fine-tuning 28
Fig. 4.8 New label data by transfer learning with modify part of data 29
Fig. 4.9 The flow chart of transfer learning with modify part of data 29
Fig. 4.10 Use original model to predict the results of general satellite imagery. 30
Fig. 4.11 New label data of transfer learning with noise 31
Fig. 4.12 The flow chart of transfer learning with noise 32
Fig. 4.13 Iterate the method of Transfer Learning with Noise 32
Fig. 4.14 Results obtained after iterate of Fig. 4.11 32
Fig. 4.15 Block-wise labeling data 33
Fig. 4.16 CNN with block-wise data 33
Fig. 4.17 Block-wise prediction 34
Fig. 4.18 The result of block-wise predicts 34
Fig. 4.19 Cut data into four blocks 35
Fig. 4.20 New label data of transfer learning with block-wise prediction 36
Fig. 4.21 The flow chart of transfer learning with block-wise prediction 36
Fig. 5.1 Each prediction result of the first category 38
Fig. 5.2 Do iteration on Transfer Learning with Modify Part of Data 39
Fig. 5.3 Each prediction accuracy of the first category 39
Fig. 5.4 Each prediction result of the second category 40
Fig. 5.5 Each prediction accuracy of the second category 40
Fig. 5.6 Do iteration on transfer learning with noise 41
Fig. 5.7 The amount of data changed to the correct label on each time 42
Fig. 5.8 Results of training the same data with many times 43
Fig. 5.9 Change block-wise label to the pixel-wise label 44
Fig. 5.10 Accuracy in the process of iteration 44
Fig. 5.11 The results we predicted on the test set after our algorithm 45

Reference
1. Chen, X.-W. and X. Lin, Big data deep learning: challenges and perspectives. IEEE access, 2014. 2: p. 514-525.
2. Simonyan, K. and A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
3. He, K., et al. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
4. Long, J., E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
5. Ronneberger, O., P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. in International Conference on Medical image computing and computer-assisted intervention. 2015. Springer.
6. Lin, G., et al. RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation. in Cvpr. 2017.
7. Pan, S.J. and Q. Yang, A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 2010. 22(10): p. 1345-1359.
8. Raina, R., et al. Self-taught learning: transfer learning from unlabeled data. in Proceedings of the 24th international conference on Machine learning. 2007. ACM.
9. Dai, W., et al. Eigentransfer: a unified framework for transfer learning. in Proceedings of the 26th Annual International Conference on Machine Learning. 2009. ACM.
10. Rolnick, D., et al., Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017.
11. Parkhi, O.M., A. Vedaldi, and A. Zisserman. Deep face recognition. in BMVC. 2015.
12. Baldi, P. and Y. Chauvin, Neural networks for fingerprint recognition. Neural Computation, 1993. 5(3): p. 402-418.
13. Jean, N., et al., Combining satellite imagery and machine learning to predict poverty. Science, 2016. 353(6301): p. 790-794.
14. Bengio, Y. Deep learning of representations for unsupervised and transfer learning. in Proceedings of ICML Workshop on Unsupervised and Transfer Learning. 2012.
15. Lee, H., et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. in Proceedings of the 26th annual international conference on machine learning. 2009. ACM.
16. Hastie, T., R. Tibshirani, and J. Friedman, Unsupervised learning, in The elements of statistical learning. 2009, Springer. p. 485-585.
17. LeCun, Y., Y. Bengio, and G. Hinton, Deep learning. nature, 2015. 521(7553): p. 436.
18. Goodfellow, I., et al., Deep learning. Vol. 1. 2016: MIT press Cambridge.
19. Zhang, S., A.E. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD. in Advances in Neural Information Processing Systems. 2015.
20. Girosi, F., M. Jones, and T. Poggio, Regularization theory and neural networks architectures. Neural computation, 1995. 7(2): p. 219-269.
21. Simard, P.Y., D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. in null. 2003. IEEE.
22. Krizhevsky, A., I. Sutskever, and G.E. Hinton. Imagenet classification with deep convolutional neural networks. in Advances in neural information processing systems. 2012.