分類問題是機器學習中很重要的一個部分，而常見的分類演算法有決策樹、k最近鄰居分類器和支撐向量機等。其中，k最近鄰居分類器是一種雖運作原理簡單但分類效果不錯、常被用在處理實際問題的一種分類器。雖k最近鄰居分類器在處理許多問題上都有不錯的比表現，但是其分類效果卻十分受到處理的資料本身的性質影響，像是資料類別的不平衡狀態、資料中的不相關特徵等。為了解決上述問題並提升k最近鄰居分類器的分類效果，本文提出使用協方差自適應調整的進化策略來最佳化k最近鄰居分類器的距離公式及投票機制，此方法稱作演化式最佳化k最近鄰居分類器(EkNN)。而最終的實驗結果顯示，最佳化距離公式中的特徵權重、特徵次方及類別權重可以幫助改善原始k最近鄰居分類器的分類效果。

Classification is a significant task in machine learning. Common classification algorithms include decision tree, k-nearest neighbors (kNN), support vector machine, and so on. The kNN algorithm is a simple yet useful classification tool in practice. However, its performance may be deeply affected by the properties of dataset, such as imbalance of classes and noisy features. To deal with these problems and enhance the performance of kNN, this study proposes optimizing the distance function and class-voting weights of kNN by the covariance matrix adaptation evolutionary strategy (CMAES). The proposed method is called evolutionary optimized kNN (EkNN). The performance of EkNN is examined on different datasets. Experimental results show that the combined evolutionary optimization on feature weights, feature powers and class weights is capable of improving the classification performance of kNN.

1 Introduction 1
2 RelatedWork 4
2.1 k-Nearest Neighbors Classifier . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Numerical Optimization . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 k-dTree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Methodology 17
3.1 Distance Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Voting Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3 Evolutionary Optimization of Distance Metric for k-Nearest Neighbors
Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 GA and CMAES for the EkNN 26
4.1 Genetic Algorithm (GA) . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2 Covariance Matrix Adaptation Evolutionary Strategy (CMAES) . . . . 28
4.3 Comparative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5 Experimental Results 33
5.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 Classification Performance . . . . . . . . . . . . . . . . . . . . . . . . 35
5.3 Speed-Up Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.4 Transformation of Feature Weights . . . . . . . . . . . . . . . . . . . . 43
6 Conclusions 52
Bibliography 54
A Classification Performance of LMNN 59

[1] A. Agapitos, M. O’Neill, and A. Brabazon. Adaptive distance metrics for nearest neighbour classification based on genetic programming. In Proceedings of European Conference on Genetic Programming, pages 1–12, 2013.
[2] T. Back. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford university press, 1996.
[3] T. Cover and P. Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1):21–27, 1967.
[4] C. Domeniconi, J. Peng, and D. Gunopulos. Locally adaptive metric nearestneighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1281–1285, 2002.
[5] C. Domeniconi and B. Yan. Nearest neighbor ensemble. In Proceedings of International Conference on Pattern Recognition, pages 228–231, 2004.
[6] S. Dudani. The distance-weighted k-nearest-neighbor rule. IEEE Transactions on Systems, Man, and Cybernetics, 6(4):325–327, 1976.
[7] J. Friedman, J. Bentley, and R. Finkel. An algorithm for finding best matches in logarithmic expected time. Association for Computing Machinery Transactions on Mathematical Software, 3(3):209–226, 1977.
[8] A. Ghosh. On optimum choice of k in nearest neighbor classification. Computational Statistics and Data Analysis, 50(11):3113–3123, 2006.
[9] D. Goldberg. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Longman, 1989.
[10] N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation, pages 312–317, 1996.
[11] J. Holland. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. University of Michigan Press, 1975.
[12] J. Holland. Genetic algorithms. Scientific American, 67(1):66–73, 1992.
[13] C. Holmes and N. Adams. A probabilistic nearest neighbour method for statistical pattern recognition. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(2):295–306, 2002.
[14] F. Hussein, N. Kharma, and R.Ward. Genetic algorithms for feature selection and weighting, a review and study. In Proceedings of Sixth International Conference on Document Analysis and Recognition, pages 1240–1244, 2001.
[15] M. jabbar, B. Deekshatulu, and P. Chandra. Classification of heart disease using knearest neighbor and genetic algorithm. Procedia Technology, 10:85 – 94, 2013.
[16] J. Kelly and L. Davis. A hybrid genetic algorithm for classification. In Proceedings of International Joint Conference on Artificial Intelligence, pages 645–650, 1991.
[17] J. Kennedy. Swarm Intelligence. Springer, 2006.
[18] L. Wang, X. Wang, and Q. Chen. Ga-based feature subset clustering for combination of multiple nearest neighbors classifiers. In Proceedings of International Conference on Machine Learning and Cybernetics, pages 2982–2987, 2005.
[19] D. Loftsgaarden and C. Quesenberry. A non-parametric estimate of a multivariate density function. Annals of Mathematical Statistics, 36:13–22, 1965.
[20] R. Mcroberts, E. NÊsset, and T. Gobakken. Optimizing the k-nearest neighbors technique for estimating forest aboveground biomass using airborne laser scanning data. Remote Sensing of Environment, 163:13–22, 2015.
[21] R. Paredes and E. Vidal. Learning weighted metrics to minimize nearest-neighbor classification error. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(7):1100–1110, 2006.
[22] W. PunchIII, E. Goodman, Min P., Lai C., P. D. Hovland, and R. Enbody. Further research on feature selection and classification using genetic algorithms. In Proceedings of International Computer Game Association, pages 557–564, 1993.
[23] I. Rechenberg. Evolutionsstrategien. Springer, 1978.
[24] H. Schwefel. Numerical Optimization of Computer Models. John Wiley & Sons, 1981.
[25] W. Siedlecki and J. Sklansky. A note on genetic algorithms for large-scale feature selection. Pattern Recognition Letters, 10(5):335 – 347, 1989.
[26] M. Tahir, A. Bouridane, and F. Kurugollu. Simultaneous feature selection and feature weighting using hybrid tabu search/k-nearest neighbor classifier. Pattern Recognition Letters, 28(4):438 – 446, 2007.
[27] S. Tan. Neighbor-weighted k-nearest neighbor for unbalanced text corpus. Expert Systems with Applications, 28(4):667 – 671, 2005.
[28] K. Weinberger and L. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10:207–244, 2009.
[29] D. Wettschereck, D. Aha, and T. Mohri. A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms. Artificial Intelligence Review, 11(1):273–314, 1997.
[30] X. Wu, V. Kumar, J. Quinlan, J. Ghosh, Q. Yang, H. Motoda, G. McLachlan, A. Ng, B. Liu, P. Yu, Z. Zhou, M. Steinbach, D. Hand, and D. Steinberg. Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1):1–37, 2008.
[31] Z. Yu, H. Chen, J. Liu, J. You, H. Leung, and G. Han. Hybrid k-nearest neighbor classifier. IEEE Transactions on Cybernetics, 46(6):1263–1275, 2016.