軟體在現代社會中所扮演的角色日趨重要。然而，為因應使用者的需求所開發出的新功能可能造成測試個案不斷地增加，進而衍生出許多問題，如:測試時間的增加或多餘、無用的測試個案等等。在過去，有許多方法被提出來解決這個問題，例如:測試個案的精簡。然而，這些技術在減少測試個案的過程中，可能會同時降低了測試套件的偵錯能力。在本論文中，我們採用模糊邏輯方法來改善現有測試個案精簡技術的問題。我們使用模糊專家系統去改良傳統四種已知的測試個案精簡演算法(HGS, GRE, GE, Greedy)以提高精簡後測試套件的偵錯能力。由於將傳統測試套件精簡技術結合模糊專家系統的研究已經有被提出，在此研究中我們將使用更多的標準以及一些重要的量測值去針對我們所提出的方法與傳統技術進行效能評估。此外，我們也更進一步的將我們的方法應用於測試個案的優先順序排列問題，並進行了簡單的實驗。我們針對9個不同大小(行碼數範圍為173-35,545)真實的程式去進行實驗，實驗結果已證明我們所提出的方法除了能夠大幅度的減少測試套件的大小外，還能同時能夠提高錯誤的偵錯能力。舉例來說，我們所提出的方法與傳統測試套件精簡演算法都有差不多的精簡套件能力；但在測試套件偵錯能力方面，Fuzzy-HGS, Fuzzy-GRE 與 Fuzzy-GE 相較於傳統的演算法分別能有平均21%, 5% 以及7% 的改善幅度。除此之外，Fuzzy-Greedy對於大的受測程式也能有很好的偵錯能力。根據所整合的理論基礎，此論文提出的方法能夠針對軟體測試及偵錯階段提出有效且實用的軟體測試方案。

Software has become increasingly important in our modern society. However, when new features are developed due to user requests, such requests could make the sizes of test-case pools bigger. Many techniques are proposed to solve this problem, such as test suite reduction. However, the ability to expose faults may be weakened when reducing the sizes of the test suites. In this thesis, we propose some methods using fuzzy logic in order to improve existing test-suite reduction techniques. The main purpose of this research is to use a Fuzzy Expert System approach in order to enhance the effectiveness of fault detection during software testing. Incorporating a Fuzzy Expert System into traditional test suite reduction techniques (HGS, GRE, GE and Greedy) is presented and studied. More objective criteria are used in order to compare the performance of our proposed and selected test suite reduction methods. Some important measures (and metrics) will also be obtained and discussed. Application of the Fuzzy Expert System approach for test case prioritization is also discussed. The experiments, based on nine real subject programs (line of code range from 173 LOCs – 35,545 LOCs), have demonstrated that the proposed approaches can significantly reduce the sizes of test suites while also improving fault detection effectiveness. For instance, Fuzzy-HGS, Fuzzy-GRE, Fuzzy-GE and Fuzzy-Greedy algorithms have almost the same reduction capability of test suite as traditional HGS, GRE, GE and Greedy algorithms. But in terms of the percentage of fault detection effectiveness loss (FDE loss), both Fuzzy-HGS, Fuzzy-GRE and Fuzzy-GE algorithms are averagely decreased by 21%, 5% and 7%, respectively. Additionally, Fuzzy-Greedy algorithm still provide the lower FDE loss for large subject programs compared to traditional Greedy algorithm. Based upon the integrated theoretical foundation, the approaches presented in this thesis offer an efficient, useful software testing scheme in the testing and debugging phases.

Abstract i
摘要 ii
Acknowledgement iii
Contents iv
List of Tables vi
List of Figures vii
Table of Notation ix
Chapter 1 Introduction 1
Chapter 2 Background and Related Works 5
Chapter 3 Review of test suite reduction techniques 13
3.1 The HGS algorithm 14
3.2 The GRE algorithm 16
3.3 The GE algorithm 18
3.4 The Greedy algorithm 19
Chapter 4 Test Suite Reduction with the Fuzzy Expert System Approach 21
4.1 FES on test suite reduction 21
4.1.1 The Fuzzification 24
4.1.2 The Fuzzy Inference 26
4.1.3 The Defuzzification 27
4.2 Fuzzy-HGS: the modified HGS algorithm by integrating FES 29
4.3 Fuzzy-GRE: the modified GRE algorithm by integrating FES 32
4.4 Fuzzy-GE: the modified GE algorithm by integrating FES 37
4.5 Fuzzy-Greedy: the modified Greedy algorithm by integrating FES 40
Chapter 5 Experimental Results and Analysis 48
5.1 Experimental setup 48
5.2 Comparison Criteria 49
5.3 Experimental results of Fuzzy-HGS 52
5.4 Experimental results of Fuzzy-GRE 57
5.5 Experimental results of Fuzzy-GE 62
5.6 Experimental results of Fuzzy-Greedy 65
5.7 Time complexity analysis 69
Chapter 6 Application: the Fuzzy Expert System Approach for Test Case Prioritization 71
6.1 Fuzzy-HGS in test case prioritization 72
6.2 Fuzzy-GRE in test case prioritization 78
6.3 Fuzzy-GE in test case prioritization 82
6.4 Fuzzy-Greedy in test case prioritization 85
6.5 Threat to validity 89
Chapter 7 Conclusions and Future Work 93
References 95

[1] “IEEE Standard for Software Test Documentation”, IEEE Standard 829-1998, Dec. 1998.
[2] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, Third Edition, MIT Press, 2009.
[3] V. Chvatal, “A Greedy Heuristic for the Set-Covering Problem”, Mathematics of Operations Research, Vol. 4, Issue 3, pp.233-235, Aug. 1979.
[4] M. J. Harrold, R. Gupta, and M. L. Soffa, “A Methodology for Controlling the Size of a Test Suite”, ACM Trans. on Software Engineering and Methodology (TOSEM), Vol. 2, Issue 3, pp.270-285, July 1993.
[5] J. W. Lin, and C. Y. Huang, “Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques”, Information and Software Technology, Vol. 51, Issue 4, pp.679-690, April 2009.
[6] C. S. Chen, “A Study of Test Suite Reduction with the Fuzzy Expert System Approach,” M. S. thesis, Department of Computer Science, National Tsing Hua University, Hsinchu, R.O.C, 2015.
[7] A. Kandel, Fuzzy Expert Systems, CRC Press, 1992.
[8] T. Y. Chen, and M. F. Lau, “A New Heuristic for Test Suite Reduction”, Information and Software Technology, Vol. 40, Issues 5-6, pp.347-354, July 1998.
[9] T. Y. Chen, and M. F. Lau, “Dividing Strategies for the Optimization of a Test Suite”, Information Processing Letters, Vol. 60, Issue 3, pp.135-141, Nov. 1996.
[10] T. Y. Chen and M. F. Lau, “A Simulation Study on Some Heuristics for Test Suite Reduction”, Information and Software Technology, Vol. 40, Issue 13, pp.777-787, Nov. 1998.
[11] T. Y. Chen and M. F. Lau, “Heuristics towards the Optimization of the Size of A Test Suite”, WIT Transactions on Information and Communications Technologies, Vol. 14, pp.415-424, 1995.
[12] S. Yoo and M. Harman, “Regression Testing Minimization, Selection and Prioritization : A Survey”, Software Testing, Verification and Reliability, Vol. 22, Issue 2, pp.67-120, March 2012.
[13] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria”, Proceedings of the 16th International Conference on Software Engineering (ICSE’94), pp. 191–200, IEEE Computer Society, CA, USA, May 1994.
[14] F. I. Vokolos, and P. G. Frankl, “Empirical Evaluation of the Textual Differencing Regression Testing Technique”, Proceedings of the 14th International Conference on Software Maintenance (ICSM 1998), pp. 44–53, Bethesda, MD, Nov. 1998.
[15] G. Rothermel, M. J. Harrold, J. von Ronne, and C. Hong, “Empirical Studies of Test-Suite Reduction”, Software Testing Verification and Reliability, Vol. 12, Issue 4, pp.219-249, Dec. 2002.
[16] W. E. Wong, J. R. Horgan, and A. P. Mathur, “Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application”, Journal of System and Software, Vol. 48, Issue 2, pp.79-89, Oct. 1999.
[17] D. Jeffrey, and N. Gupta, “Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction”, IEEE Transactions on Software Engineering, Vol. 33, Issue 2, pp.108-123, Feb. 2007.
[18] M. P. E. Heimdahl and D. George, “Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing”, Proceedings of the 19th IEEE International Conference on Automated Software Engineering (ASE 2004), pp. 176–185, USA, September 2004.
[19] S. Sampath, and R. C. Bryce, “Improving the Effectiveness of Test Suite Reduction for User- Session-Based Testing of Web Applications”, Information and Software Technology, Vol. 54, Issue 7, pp.724-738, July 2012.
[20] N. Mansour and K. El-Fakih, “Simulated Annealing and Genetic Algorithms for Optimal Regression Testing”, Journal of Software Maintenance, Vol. 11, Issue 1, pp.19-34, 1999.
[21] M. Harder, J. Mellen, and M.D. Ernst, “Improving Test Suites via Operational Abstraction”, Proceedings of the 25th International Conference on Software Engineering (ICSE 2003), pp. 60–71, DC, USA, May 2003.
[22] L. Zhang, D. Marinov, L. Zhang, and S. Khurshid, “An Empirical Study of JUnit Test-Suite Reduction”, Proceedings of 22nd IEEE International Symposium on Software Reliability Engineering (ISSRE 2011), pp. 170–179, Hiroshima, Nov. 2011.
[23] A. Schwartz and H. Do, “A Fuzzy Expert System for Cost-Effective Regression Testing Strategies”, Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM 2013), pp. 1–10, Eindhoven, Sept. 2013.
[24] A. M. Smith, J. Geiger, G. M. Kapfhammer, and M. L. Soffa, “Test Suite Reduction and Prioritization with Call Trees”, Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), pp.539–540, NY, USA, Nov. 2007.
[25] C. Murphy, Z. Zoomkawalla, and K. Narita, “Automatic Test Case Generation and Test Suite　Reduction for Closed-Loop Controller Software”, Technical Report, Dept. of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA, Jan. 2013.
[26] H Mei, D Hao, L Zhang, L Zhang, J Zhou, and G Rothermel, “A Static Approach to Prioritizing JUnit Test Cases”, IEEE Transactions on Software Engineering, Vol. 38, Issue 6, pp.1258-1275, NJ, USA, Nov. 2012.
[27] L. Zhang, S. S. Hou, C. Guo, T. Xie and H. Mei, “Time-Aware Test-Case Prioritization using Integer Linear Programming”, Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA 2009), pp. 213–224, New York, USA, July 2009.
[28] T. Yu, W. Srisa-an, and G. Rothermel, “SimRT: An Automated Framework to Support Regression Testing for Data Races”, Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), pp. 48–59, New York, USA, May/June 2014.
[29] A. Gotlieb and D. Marijan, “FLOWER: Optimal Test Suite Reduction as a Network”, Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014), pp. 171–180, New York, USA, July 2014.
[30] A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov, “Balancing Trade-Offs in Test-Suite Reduction”, Proceedings of the 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE 2014), pp. 246–256, New York, USA, Nov. 2014.
[31] C. T. Lin, K. W. Tang, and G. M. Kapfhammer, “Test Suite Reduction Methods that Decrease Regression Testing Costs by Identifying Irreplaceable Tests”, Information and Software Technology, Vol. 56, Issue 10, pp.1322-1344, Oct. 2014.
[32] Bo Xie, Requirement Based Regression Test Suite Reduction Using Dependence Analysis, Master Thesis, School of Information Technology and Engineering, University of Ottawa, Ottawa, Ontario, Canada, 2005. Available at: http://hdl.handle.net/10393/27200.
[33] Y. Chen, R. L. Probert, and H. Ural, “Regression Test Suite Reduction Using Extended Dependence Analysis”, In Proceeding of the 4th International Workshop on Software Quality Assurance (SOQUA '07): in conjunction with the 6th ESEC/FSE joint meeting, pp. 62–69, New York, USA, Sept. 2007.
[34] A. González-Sanchez, E. Piel, H. G. Gross, A. J. C. van Gemund, “Prioritizing Tests for Software Fault Localization”, Proceedings of the 10th International Conference on Quality Software (QSIC), pp.42–51, Zhangjiajie, July 2010.
[35] A. González-Sanchez, R. Abreu, H. G. Gross, A. J. C. van Gemund, “Spectrum-Based Sequential Diagnosis”, Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence (AAAI 2011), pp. 189–196, San Francisco, August 2011.
[36] A. González-Sanchez, É. Piel, R. Abreu, H. G. Gross, A. J. C. van Gemund, “Prioritizing Tests for Software Fault Diagnosis”, Journal of Software: Practice & Experience, Vol. 41, Issue 10, pp. 1105–1129, Sept. 2011.
[37] A. González-Sanchez, R. Abreu, H. G. Gross, A. J. C. van Gemund, “RAPTOR: Greedy Diagnostic Prioritization by Ambiguity Group Reduction”, Proceedings of the 22nd International Workshop on Principles of Diagnosis (DX'11), pp. 84-91, Marnau, Germany, Oct. 2011.
[38] A. González-Sanchez, R. Abreu, H. G. Gross, A. J. C. van Gemund, “Prioritizing Tests for Fault Localization through Ambiguity Group Reduction”, Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pp.83–92, Lawrence, KS, Nov. 2011.
[39] A. Gonzalez-Sanchez, E. Piel, R. Abreu, H-G. Gross, A.J.C. van Gemund, “Prioritizing Tests for Fault Localization”, Situation Awareness with Systems of Systems, P. van de Laar, J. Tretmans, M. Borth (eds.), Springer, pp. 251–261, New York, 2012.
[40] B. Jiang, W. K. Chan, T. H. Tse, “On Practical Adequate Test Suites for Integrated Test Case Prioritization and Fault Localization”, Proceedings of the 11th International Conference on Quality Software (QSIC 2011), pp. 21–30, Madrid, Spain, July 2011.
[41] D. Hao, L. Zhang, X. Wu, H. Mei, G. Rothermel, “On-Demand Test Suite Reduction”, Proceedings of 34th International Conference on Software Engineering (ICSE 2012), pp. 738–748, NJ, USA, June 2012.
[42] G. Fraser and F. Wotawa, “Redundancy Based Test-Suite Reduction”, Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE'07), pp. 291–305, Dublin, Ireland, March/April 2007.
[43] A. Gotlieb and D. Marijan, “FLOWER: Optimal Test Suite Reduction as A Network Maximum Flow”, Proceeding of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014), pp. 171–180, CA, USA, July 2014.
[44] L. Zadeh, “Fuzzy sets”, Information and Control, Vol. 8, Issue 3, pp.238-353, June 1965.
[45] E. H. Mamdani, and S. Assilian, “An Experiment in Linguistic Synthesis with a Fuzzy Logic Controller”, International Journal of Man-Machine Studies, Vol. 7, Issue 1, pp.1-13, Jan. 1975.
[46] T. Takagi, and M. Sugeno, “Fuzzy Identification of Systems and Its Applications to Modeling and Control”, IEEE Transactions on Systems, Man and Cybernetics, Vol. SMC-15, Issue 1, pp.116-132, Jan.-Feb. 1985.
[47] P. C. Shill, K. K. Pal, M. F. Amin, and K. Murase, “Genetic Algorithm Based Fully Automated and Adaptive Fuzzy Logic Controller”, Proceedings of IEEE International Conference on Fuzzy Systems, pp. 1572–1579, Taipei, June 2011.
[48] P. R. Srivastava, S. Kumar, A. Singh, and G. Raghurama, “Software Testing Effort: An Assessment through Fuzzy Criteria Approach”, Journal of Uncertain Systems, Vol. 5, Issue 3, pp.183-201, 2011.
[49] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold, “Prioritizing Test Cases for Regression Testing”, IEEE Transactions on Software Engineering, Vol. 27, Issue 10, pp.929-948, 2001.
[50] M. R. Garey, and D. S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, Freeman, New York, 1979.
[51] G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong, “An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites”, Proceedings of the 14th International Conference on Software Maintenance (ICSM 1998), pp. 34–43, Bethesda, MD, Nov. 1998.
[52] H. Zhong, L. Zhang, and H. Mei, “An Experimental Study of Four Typical Test Suite Reduction Techniques”, Information and Software Technology. Vol. 50, Issue 6, pp.534-546, 2008.
[53] S. Tallam and N. Gupta, “A Concept Analysis Inspired Greed Algorithm For Test Suite Minimization”, Proceedings of the 6th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE 2005), co-located with ESEC/FSE 2005 Conference, pp. 35–42, NY, USA, Sept. 2005.
[54] S. Sprenkle, E. Sampath, A. Gibson, L. Pollock, and A. Souter, “An Empirical Comparison of Test Suite Reduction Techniques for User-Session-Based Testing of Web Applications”, Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM 2005), pp. 587–596, Budapest, Hungary, Sept. 2005.
[55] D. Jeffrey, and N. Gupta, “Test Suite Reduction with Selective Redundancy”, Proceedings of the 21st International Conference on Software Maintenance (ICSM 2005), pp. 549–558, Budapest, Hungary, Sept. 2005.
[56] Elmer Dadios (Ed.), Fuzzy Logic - Controls, Concepts, Theories and Applications, InTech, DOI: 10.5772/36321, 2012.
[57] D. Driankov, H. Hellendoorn and M. Reinfrank, An Introduction to Fuzzy Control, Second Edition, Springer-Verlag, Berlin, 1996.
[58] Y. Bai and D. Wang, “Fundamentals of Fuzzy Logic Control – Fuzzy Sets Fuzzy Rules and Defuzzification”, In Adcanced Fuzzy Loigc Technologies in Industrial Applications, p.17-36, 2006.
[59] S. Tallam, and N. Gupta, “A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization”, Proceedings of the 6th Workshop Program Analysis for Software Tools and Engineering, pp. 35–42, NY, USA, Jan. 2005.
[60] S. Sampath, S. Sprenkle, E. Gibson, and L. Pollock, “Integrating Customized Test Requirements with Traditional Requirements in Web Application Testing”, Proceedings of the Workshop on Testing, Analysis, and Verification of Web Services and Applications (TAV-WEB 2006), Portland, Maine, pp. 23–32, 2006.
[61] J. von Ronne, Test Suite Minimization: An Empirical Investigation, University Honors College Thesis, Oregon State University, Corvallis, USA, 1999.
[62] The Software-Artifact Infrastructure Repository, 2008. Available from http://sir.unl.edu/portal/.
[63] P. C. Jorgensen, Software Testing: A Craftsman's Approach, 4th Edition, CRC Press, 2013.
[64] N. Fenton and J. Bieman, Software Metrics: A Rigorous and Practical Approach, 3rd Edition, CRC Press, 2015.
[65] G. Paliouras, Scalability of Machine Learning Algorithms, M.Sc Thesis, Department of Computer Science, University of Manchester, UK, Nov. 1993.
[66] V. P. Kumar and A. Gupta, “Analyzing Scalability of Parallel Algorithms and Architectures”, Journal of Parallel and Distributed Computing–Special Issue on Scalability of Parallel Algorithms and Architectures, Vol. 22, Issue 3, pp.379-391, 1994.
[67] X. H. Sun and D. T. Rover, “Scalability of Parallel Algorithm–Machine Combinations”, IEEE Transactions on Parallel and Distributed System, Vol. 5, Issue 6, pp.599-613, 1994.
[68] G. Keller, and B. Warrack, Statistics for Management and Economics, Thomson, 2003.
[69] S. D. Conte, H. E. Dunsmore, and V. Y. Shen, Software Engineering Metrics and Models, Benjamin-Cummings Publishing Co., Inc., Redwood City, CA, 1986.
[70] B. A. Kitchenham, L. M. Pickard, S. G. MacDonell, and M. J. Shepperd, “What Accuracy Statistics Really Measure”, IEE Proceedings－Software, Vol. 148, Issue 3, pp.81-85, 2001.
[71] T Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit, “A Simulation Study of the Model Evaluation Criterion MMRE”, IEEE Transactions on Software Engineering, Vol. 29, Issue 11, pp.985-995, 2003.
[72] Z. Li, M. Harman, and R. M. Hierons, “Search Algorithms for Regression Test Case Prioritization”, IEEE Transactions on Software Engineering, Vol. 33, Issue 4, pp.225-237, 2007.
[73] J. J. Chilenski, and P. H. Newcomb, “Formal Specification Tools for Test Coverage Analysis”, Proceedings of the 9th Knowledge-Based Software Engineering Conference, pp. 59-68, Monterey, CA, Sept. 1994.
[74] G. Rothermel, R. Untch, C. Chu, and M. J. Harrold, “Test Case Prioritization: An Empirical Study”, Proceedings of of the International Conference on Software Maintenance (ICSM 1999), pp.179–188, Oxford, Sept. 1999.
[75] S. Elbaum, A. Malishevsky, and G. Rothermel, “Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization”, Proceedings of the 23rd International Conference on Software Engineering (ICSE 2001), pp.329–338, Toronto, ON, Canada, May 2001.
[76] C. Y. Huang, J. R. Chang, and Y. H. Chang, “Design and Analysis of GUI Test Case Prioritization Using Weight-Based Methods”, Journal of Systems and Software, Vol. 83, Issue 4, pp.646-659, 2010.
[77] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering, Berlin; New York: Springer, 2012.
[78] S. Tallam and N. Gupta, “A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization”, Proceedings of the 6th ACM SIGPLAN-SIGSOFT Workshop on Program analysis for Software Tools and Engineering, Co-located with ESEC/FSE 2005 (PASTE 2005), pp. 35–42, New York, 2005.
[79] M. Pezze and M. Young, Software Testing and Analysis : Process, Principles and Techniques, John Wiley & Sons, Inc, New York, NY, USA, 2007.