This research focuses on botnet detection through implementation of techniques such as traffic analysis, unsupervised machine learning, and similarity analysis between benign traffic data and bot traffic data. In this study, we tested and experimented with different clustering algorithms and recorded their accuracy with our prepared datasets. Later, the best clustering algorithm was used to proceed with the next steps of the methodology such as determination of majority clusters (cluster with most flows), removal of duplicate flows, and calculation of similarity analysis. Results were recorded for the removal of duplicate flows stage, the results indicate how many flows each majority cluster contains and how many duplicate flows were removed from this majority cluster. Next, results for similarity analysis indicate the value of the similarity coefficient for the comparisons between all datasets (bot datasets and benign dataset.) With these results we can conclude and present some concluding heuristics for determining possible bot infection in a certain host.

本研究使用了不同的技術來偵測殭屍網路, 我們使用了網路流量分析, 非監督式學習, 以及分析正常網路與殭屍網路之間的相似性等技術來實踐。
研究中, 我們測試了不同的分群演算法並比較它們的表現, 下一步,我們選擇表現最好的分群演算法,去決定主群體還有移除多餘且相同的網路資料, 並分析其相似度。
藉由計算出的網路相似度結果, 我們設計出了啟發式的方法來偵測殭屍網路

Contents
Declaration of Authorship i
Abstract ii
Acknowledgements iii
List of Figures vi
List of Tables vii
1 Introduction 1
2 Background 2
2.1 Defining and Understanding a Botnet . . . . . . . . . . . . . . . . . . . . 2
2.1.1 Botnet Life-Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.1.1 Infection Phase . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.1.2 Communication Phase . . . . . . . . . . . . . . . . . . . . 3
2.1.1.3 Attack Phase . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Command and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.1 Centralized Botnets . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.2 Distributed Botnets . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.3 Hybrid Botnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Unsupervised Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 Comparison Between Unsupervised and Supervised Learning . . . 7
3 Botnet Detection and Unsupervised Machine Learning 8
3.1 Traffic Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Simple K-Means Clustering . . . . . . . . . . . . . . . . . . . . . . 9
3.2.2 EM Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Clustering & Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4 Clustering Evaluation in Weka . . . . . . . . . . . . . . . . . . . . . . . . 11
4 Botnet Detection Datasets 12
4.1 Benign P2P, Zeus, and Waledac Botnet Traces . . . . . . . . . . . . . . . 12
iv
Contents v
4.1.1 Dataset Preparations . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Storm Bot Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.1 Storm Dataset 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.2 Storm Dataset 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 Methodology 15
5.1 Packet Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.2 Flow Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2.1 Flow Features and Statistics . . . . . . . . . . . . . . . . . . . . . . 16
5.3 Data Pre-processing: Data Labeling . . . . . . . . . . . . . . . . . . . . . 17
5.4 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.4.1 Information Gain Attribute Evaluation . . . . . . . . . . . . . . . . 19
5.5 Importing Data Into Weka . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.6 Flow Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.6.1 Classes to Clusters Evaluation . . . . . . . . . . . . . . . . . . . . . 20
5.7 Determining Majority Clusters . . . . . . . . . . . . . . . . . . . . . . . . 22
5.8 Removal of Duplicate Flows . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.9 Similarity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6 Experiments and Results 24
6.1 Feature Selection & Attribute Evaluation Results . . . . . . . . . . . . . . 24
6.2 Classes to Clusters Evaluation Results . . . . . . . . . . . . . . . . . . . . 26
6.2.1 Performance and Accuracy Results . . . . . . . . . . . . . . . . . . 27
6.3 Duplicate Flow Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 Similarity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
7 Conclusions 30
7.1 Concluding Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7.1.1 Similarity Analysis and Unknown Botnets . . . . . . . . . . . . . . 30
7.1.2 Possible Improvements for Future Work . . . . . . . . . . . . . . . 31
8 Related Work 32
8.1 BotOnus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
8.2 Botminer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
8.3 Peershark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
A Complete List of Features 35
Bibliography

[1] Matija Stevanovic and Myrup Pedersen. An efficient flow-based botnet detection
using supervised machine learning. Networking and Security Section, Department
of Electronic Systems, .
[2] Shang-Chiuan Su. Detecting p2p botnet in software defined network. Institute of
Network Engineering College of Computer Science National Chiao Tung University.
[3] Sharath Chandra Guntuku, Pratik Narang, and Chittaranjan Hota. Real-time peerto-
peer botnet detection framework based on bayesian regularized neural network.
Institute of Network Engineering College of Computer Science National Chiao Tung
University.
[4] Pratik Narang1, Chittaranjan Hota1, and VN Venkatakrishnan. Peershark: flowclustering
and conversation-generation for malicious peer-to-peer traffic identification.
EURASIP Journal on Information Security, 2014.
[5] Charles A. Fowler and Robert J. Hammell II. Converting pcaps into weka mineable
data. Department of Computer and Information Sciences Towson University.
[6] John Brozycki. Capturing and analyzing packets with perl. SANS Institute InfoSec
Reading Room.
[7] Daniel Arndt. How to: Calculating flow statistics using netmate. URL http:
//dan.arndt.ca/nims/calculating-flow-statistics-using-netmate/.
[8] Anthony McGregor, Mark Hall, Perry Lorier, and James Brunskill. Flow clustering
using machine learning techniques. The University of Waikato, New Zealand.
[9] Sherif Saad, Issa Traore, Ali A. Ghorbani, Bassam Sayed, David Zhao, Wei Lu, John
Felix, and Payman Hakimian. Detecting p2p botnets through network behavior
analysis and machine learning. Proceedings of 9th Annual Conference on Privacy,
Security and Trust (PST2011), 2011.
[10] Khalid Huseynov, Kwangjo Kim, and Paul D. Yoo. Semi-supervised botnet detection
using ant colony clustering. Cryptography and Information Security, 2014.
39
Bibliography 40
[11] David Zhao, Issa Traore, Bassam Sayed, and ... Botnet detection based on traffic
behavior analysis and flow intervals. Elsevier, 2013.
[12] Tao Cai and Futai Zou. Detecting http botnet with clustering network traffic.
[13] Chittaranjan Hota Pratik Narang, Jagan Mohan Reddy. Feature selection for detection
of peer-to-peer botnet traffic.
[14] Pijush Barthakur, Manoj Dahal, and Mrinal Kanti Ghose. Clusibothealer: Botnet
detection through similarity analysis of clusters. Journal of Advances in Computer
Networks, Vol. 3, No. 1, March 2015.
[15] Nicolás García-Pedrajas, Aida de Haro-García, and Javier Pérez-Rodríguez. A scalable
approach to simultaneous evolutionary instance and feature selection. Elsevier.
[16] Matija Stevanovic and Jens Myrup Pedersen. An efficient flow-based botnet detection
using supervised machine learning. Networking and Security Section, Depart-
ment of Electronic Systems, Aalborg University, .
[17] Paul Barford and Vinod Yegneswaran. An inside look at botnets. Computer Sciences
Department University of Wisconsin, Madison.
[18] Kamaldeep Singh, Sharath Chandra Guntuku, Abhishek Thakur, and Chittaranjan
Hota. Big data analytics framework for peer-to-peer botnet detection. Elsevier.
[19] Guofei Gu, Roberto Perdisci, Junjie Zhang, and Wenke Lee. Botminer: Clustering
analysis of network traffic for protocol- and structure-independent botnet detection.
College of Computing, Georgia Institute of Technology.
[20] . Botnet detection and collapse based on traffic analysis. , .
[21] David Zhao. Botnet detection based on traffic behavior analysis and flow intervals.
Elsevier.
[22] Mosa Yahyazadeh and Mahdi Abadi. Botonus: An online unsupervised method for
botnet detection. The ISC Int’l Journal of Information Security.
[23] Wei Lu, Goaletsa Rammidi, and Ali A. Ghorbani. Clustering botnet communication
traffic based on n-gram feature selection. Elsevier.
[24] Norbert Pohlmanna Christian J. Dietricha, Christian Rossowa. Cocospot: Clustering
and recognizing botnet command and control channels using traffic analysis.
Elsevier.
[25] Esraa Alomari and Selvakumar Manickama. Design, deployment and use of httpbased
botnet testbed. National Advanced IPv6 Centre (NAv6), Universiti Sains
Malaysia, Malaysia.
Bibliography 41
[26] Pratik Narang, Jagan Mohan Reddy, and Chittaranjan Hota. Feature selection
for detection of peer-to-peer botnet traffic. Department of Computer Science &
Engineering Birla Institute of Technology and Science-Pilani.
[27] Stevanovic Matija and Pedersen Jens Myrup. Machine learning for identifying botnet
network traffic. Department of Computer Science & Engineering Birla Institute of
Technology and Science-Pilani.
[28] Vaibhav Nivargi. Machine learning based botnet detection.
[29] Babak Rahbarinia and Roberto Perdisci. Peerrush: Mining for unwanted p2p traffic.
Dept. of Computer Science, University of Georgia.
[30] Khalid Huseynov and Kwangjo Kim. Unsupervised hadoop-based p2p botnet detection
with threshold setting. Department of Computer Science, Korea Advanced
Institute of Science and Technology.
[31] Zoubin Ghahramani. Unsupervised learning. Gatsby Computational Neuroscience
Unit University College London, UK.
[32] Carl Livadas. Using machine learning techniques to identify botnet traffic. Inter-
network Research Department BBN Technologies.
[33] Lasse Trolle Borup. Peer-to-peer botnets: A case study on waledac. Kongens Lyngby
2009.