軟體已成為我們日常生活中重要的一部分。根據使用者的需求，開發者會開發新的功能，為了確保軟體的品質是良好的，必須產生新的測試個案並加入現有的測試個案池當中，為此，測試個案池的大小往往變得過大以致於無法徹底執行所有測試個案，而過大的測試個案池需要花費大量時間導致產生沒有效率的回歸測試。測試個案精簡法是常見於藉著移除冗餘的測試個案以解決測試套件過大的問題。藉著使用測試個案精簡法，可有效地降低測試套件大小並同時保持與原始測試池相同的覆蓋率。然而，部分現存的測試個案精簡法只考慮一種測試標準，若我們能夠只用一種以上的測試標準，將可以提升偵錯的能力；有另一部分的方法使用兩種測試標準，但是直到首要標準遇上相同而無法決定選取哪一個測試個案時，才會使用次要的標準。本研究最主要的目標在於使用四種修改後的權重組合演算法來改善偵錯效率並同時且彈性地使用兩種不同類型的測試標準。除此之外，我們也討論如何將基因演算法合併進我們的方法來求得兩種測試標準的最佳權重係數分配。實驗結果指出我們提出的權重組合演算法中，有77.5%的方法於套件規模縮減持平或是更好；83.75%的方法於偵錯效率上持平或更好；82.5%的方法於錯誤與測試個案比例上持平或更好。

Software has become an important part of our daily lives. Due to user requests, new features are developed. To ensure the quality of a software, new test cases have to be generated and included in the current test pool. Further, the size of the test pool will often grow too large to test through all of the test cases. A very large test pool will take large amounts of time to produce inefficient regression testing. Test suite reduction is typically used to solve the test suite size problem by removing redundant test cases. By using test suite reduction, the test pool size can be efficiently reduced, while the remaining test cases are still able to provide the same coverage as the original test pool. However, some of the existing test suite reduction methods consider only one testing criterion. If we use more than one testing criterion, we can improve the capability of fault detection effectiveness. Also, some people consider two testing criteria; however, they don’t use the second criterion until there is a tie situation in the primary criterion. The main purpose of this research is to use 4 modified weighted combinatorial algorithms to improve the effectiveness of fault detection, and to consider two different types of testing criteria, flexibly and simultaneously. Further, we also discuss how to incorporate a genetic algorithm to find the best weighting factor value assignments for each testing criterion. Experimental results show that 77.5% of our proposed weighted combinatorial algorithms have an even or better performance in suite size reduction, 83.75% of our approaches perform even or better in fault detection effectiveness, and 82.5% of our approaches have even or better effectiveness in fault-to-test ratio.

Abstract in Chinese i
Abstract ii
Acknowledgement iii
Contents iv
List of Tables vi
List of Figures vii
Table of Notations viii
Chapter 1 Introduction 1
Chapter 2 Related Works 7
Chapter 3 Review of Test Suite Reduction Techniques 14
3.1 The HGS Algorithm 14
3.2 The GRE Algorithm 16
3.3 The GE Algorithm 18
3.4 The Greedy Algorithm 20
3.5 Traditional Algorithms Discussion 20
Chapter 4 Weighted Combinatorial Algorithms 24
4.1 Weighted Statistical Means 24
4.2 Evaluation Metric 27
4.3 Proposed Weighted Combinatorial Algorithms 35
4.3.1 WC-HGS: The Modified Weighted Combinatorial HGS Algorithm 35
4.3.2 WC-GRE: The Modified Weighted Combinatorial GRE Algorithm 38
4.3.3 WC-GE: The Modified Weighted Combinatorial GE 42
4.3.4 WC-Greedy: The Modified Weighted Combinatorial Greedy Algorithm 47
4.4 Weighting Factor Assignment 49
Chapter 5 Experiments and Analysis 52
5.1 Experimental Setup and Subject Programs 52
5.2 Comparison Criteria 56
5.3 Experimental Results of Applying the Weighted Combinatorial Algorithms 59
5.3.1 Suite Size Reduction 59
5.3.2 Fault Detection Effectiveness Loss 62
5.3.3 A t-test for Matched Pairs 65
5.3.4 Fault-to-test Ratio 68
5.3.5 Execution Time 70
5.3.6 Percentage of Tie Occurrence 72
5.4 Discussions and Observations 73
5.4.1 The size of subject programs 73
5.4.2 The randomness existing in traditional algorithms 74
5.4.3 The use of the FDE Loss metric 74
5.4.4 The implementation of weighted statistical means 74
5.5 Time Complexity Analysis 76
5.6 Threat to Validity 77
Chapter 6 Tri-Criteria Binary ILP Model 82
6.1. Evaluation Metric 82
6.2. Proposed Tri-Criteria binary ILP Model 83
6.3. Experimental Results of Tri-Criteria Binary ILP Model 87
Chapter 7 Conclusions and Future Works 91
References 94
Appendix 101

[1] P. Ammann and J. Offutt. Introduction to Software Testing, Cambridge University Press, 2008.
[2] D. Binkley, “Semantics Guided Regression Test Cost Reduction,” IEEE Transactions on Software Engineering, Vol. 23, No. 8, pp. 498–516, Aug. 1997.
[3] S. Yoo and M. Harman, “Regression Testing Minimisation, Selection and Prioritization: A Survey,” Software Testing, Verification & Reliability, Vol. 22, Issue 2, pp. 67–120, Mar. 2012.
[4] M. J. Harrold, R. Gupta, and M. L. Soffa, “A Methodology for Controlling the Size of a Test Suite,” ACM Transactions on Software Engineering & Methodology (TOSEM), Vol. 2, Issue 3, pp. 270–285, Jul. 1993.
[5] H. Y. Hsu and A. Orso, “MINTS: A General Framework and Tool for Supporting Test-Suite Minimization,” Proceedings of the 31st IEEE International Conference on Software Engineering, Vancouver, BC, pp. 419–429, May 2009.
[6] D. Jeffrey and N. Gupta, “Test Suite Reduction with Selective Redundancy,” Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM 2005), pp. 549–558, Sept. 2005.
[7] J. W. Lin and C. Y. Huang, “Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques,” Information & Software Technology, Vol. 51, Issue 4, pp. 679–690, Apr. 2009.
[8] W. Masri and M. El-Ghali, “Test Case Filtering and Prioritization Based on Coverage of Combinations of Program Elements,” Proceedings of the 7th International Workshop on Dynamic Analysis (WODA ’09), Chicago, IL, pp. 29–34, Jul. 2009.
[9] C. T. Lin, K. W. Tang, and G. M. Kapfhammer, “Test Suite Reduction Methods that Decrease Regression Testing Costs by Identifying Irreplaceable Tests,” Information & Software Technology, Vol. 56, Issue 10, pp. 1322–1344, Oct. 2014.
[10] S. Sampath and R.C. Bryce, “Improving the Effectiveness of Test Suite Reduction for User-Session-Based Testing of Web Applications,” Information & Software Technology, Vol. 54, Issue 7, pp. 724–738, Jul. 2012.
[11] H. Zhu, “A Formal Analysis of the Subsume Relation between Software Test Adequacy Criteria,” IEEE Transactions on Software Engineering, Vol. 22, No. 4, pp. 248–255, Apr. 1996.
[12] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold, “Prioritizing Test Cases for Regression Testing,” IEEE Transactions on Software Engineering, Vol. 27, No 10, pp. 929–948, Oct. 2001.
[13] H. Zhong, L. Zhang and H. Mei, “An Experimental Study of Four Typical Test Suite Reduction Techniques,” Information & Software Technology, Vol. 50, Issue 6, pp. 534–546, May 2008.
[14] D. Jeffrey and N. Gupta, “Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction,” IEEE Transactions on Software Engineering, Vol. 33, No. 2, pp. 108–123, Feb. 2007.
[15] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-based Test Adequacy Criteria,” Proceedings of the 16th International Conference on Software Engineering (ICSE ‘94), CA, USA, pp. 191–200, May 1994.
[16] C. S. Chen, “A Study of Test Suite Reduction with the Fuzzy Expert System Approach,” M. S. thesis, Department of Computer Science, National Tsing Hua University, Hsinchu, R.O.C, 2015.
[17] C. Y. Chiu, “Analysis and Comparison of Weighted Combinatorial Algorithms for Test Suite Reduction,” M. S. thesis, Department of Computer Science, National Tsing Hua University, Hsinchu, R.O.C, 2015.
[18] T. Y. Chen and M. F. Lau, “Heuristics Towards the Optimization of the Size of A Test Suite,” WIT Transactions on Information and Communications Technologies, Vol. 14, pp. 415–424, 1995.
[19] T. Y. Chen and M. F. Lau, “A Simulation Study on Some Heuristics for Test Suite Reduction,” Information & Software Technology, Vol. 40, Issue 13, pp 777–787, Nov. 1998.
[20] V. Chvatal, “A Greedy Heuristic for the Set-Covering Problem,” Mathematics of Operations Research, Vol. 4, No. 3, pp. 233–235, Aug. 1979.
[21] F. I. Vokolos, and P. G. Frankl, “Empirical Evaluation of the Textual Differencing Regression Testing Technique,” Proceedings of the 14th International Conference on Software Maintenance, Bethesda, MD, pp. 44–53, Nov. 1998.
[22] J. Black, E. Melachrinoudis, and D. Kaeli, “Bi-Criteria Models for All-Uses Test Suite Reduction,” Proceedings of the 26th International Conference on Software Engineering, pp. 106–115, May 2004.
[23] W. E. Wong, J. R. Horgan, and A. P. Mathur, “Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application,” Proceedings of the 21st Annual International Computer Software and Applications Conference (COMPSAC ‘97), Washington, DC, pp. 522–528, Aug. 1997.
[24] M. P. E. Heimdahl and D. George, “Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing,” Proceedings of the 19th International Conference on Automated Software Engineering, pp. 176–185, Sept. 2004.
[25] C. Zhang, A. Groce, and M. A. Alipour, “Using Test Case Reduction and Prioritization to Improve Symbolic Execution,” Proceedings of the 2014 International Symposium on Software Testing and Analysis, pp. 160–170, Jul. 2014.
[26] N. Mansour and K. El-Fakih, “Simulated Annealing and Genetic Algorithms for Optimal Regression Testing,” Journal of Software Maintenance: Research and Practice, Vol. 11, Issue 1, pp. 19–34, Jan. –Feb. 1999.
[27] M. Harder, J. Mellen, and M. D. Ernst, “Improving Test Suites via Operational Abstraction”, Proceedings of the 25th International Conference on Software Engineering, pp. 60–71, May 2003.
[28] L. Zhang, D. Marinov, L. Zhang, and S. Khurshid, “An Empirical Study of JUnit Test-Suite Reduction”, Proceedings of the 22nd International Symposium on Software Reliability Engineering, Hiroshima, pp. 170–179, Nov. 2011.
[29] A. E. V. B. Coutinho, E. G. Cartaxo, and P. Duarte, “Analysis of Distance Functions for Similarity-Based Test Suite Reduction in the Context of Model-Based Testing,” Software Quality Journal, Vol. 24, Issue 2, pp. 407–445, Dec. 2014.
[30] G. Rothermel, M. J. Harrold, J. von Ronne and C. Hong, “Empirical Studies of Test-Suite Reduction,” Software Testing, Verification & Reliability, Vol. 12, Issue 4, pp. 219–249, Dec. 2002.
[31] A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov, “Balancing Trade-Offs in Test-Suite Reduction,” Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, Hong Kong, China, pp. 246–256, Nov. 2014.
[32] D. Di Nardo, N. Alshahwan, L. Briand, and L. Briand, “Coverage-Based Regression Test Case Selection, Minimization and Prioritization: A Case Study on An Industrial System,” Software Testing, Verification and Reliability, Vol. 25, Issue 4, pp. 371–396, June 2015.
[33] S. Mirarab, S. Akhlaghi, and L. Tahvildari, “Size-Constrained Regression Test Case Selection Using MultiCriteria Optimization,” IEEE Transactions on Software Engineering, Vol. 38, No. 4, pp. 936–956, Jul./Aug. 2012.
[34] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold, “Prioritizing Test Cases for Regression Testing,” IEEE Transactions on Software Engineering, Vol. 27, No. 10, pp. 929–948, Oct. 2001.
[35] L. Zhang, S. S. Hou, C. Guo, T. Xie and H. Mei, “Time-Aware Test-Case Prioritization Using Integer Linear Programming,” Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA ’09), Chicago, IL, pp. 213–224, Jul. 2009.
[36] S. Yoo and M. Harman, “Using Hybrid Algorithm for Pareto Efficient Multi-Objective Test Suite Minimization,” Journal of Systems and Software, Vol. 83, Issue 4, pp 689–701, Apr. 2010.
[37] A. Panichella and M. Di Penta, “Improving Multi-Objective Test Case Selection by Injecting Diversity in Genetic Algorithms,” IEEE Transactions on Software Engineering, vol. 41, pp. 358–383, Oct. 2015.
[38] Y. C. Huang, C. Y. Huang, J. R. Chang, and T. Y. Chen, “Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History,” Proceedings of the 34th IEEE Annual Computer Software and Applications Conference, Seoul, pp. 413–418, Jul. 2010.
[39] K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. Roos, “Time-Aware Test Suite Prioritization,” Proceedings of the 2006 International Symposium on Software Testing and Analysis, pp. 1–12, Jul. 2006.
[40] S. J. Huang and N. H. Chiu, “Optimization of Analogy Weights by Genetic Algorithm for Software Effort Estimation,” Information & Software Technology, Vol. 48, Issue 11, pp. 1034–1045, Nov. 2006.
[41] H. Y. Jiang, C. K. Chang, D. Zhu, and S. Cheng, “A Foundational Study on the Applicability of Genetic Algorithm to Software Engineering Problems,” Proceedings of 2007 IEEE Congress Evolutionary Computation (CEC 2007), Singapore, pp. 2210–2219, Sep. 2007.
[42] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley, New York, USA, 1989.
[43] M. Mitchell, An Introduction to Genetic Algorithms, The MIT Press, Cambridge, MA, USA, 1998.
[44] C. J. Hsu and C. Y. Huang, “Optimal Weighted Combinational Models for Software Reliability Estimation and Analysis,” IEEE Transactions on Reliability, Vol. 63, No. 3, pp. 731–749, Sep. 2014.
[45] R. M. Karp. Reducibility among Combinatorial Problems, Complexity of Computer Computations, Plenum Press, pp. 85–103, 1972.
[46] H. Zhong, L. Zhang, and H. Mei, “An Experimental Comparison of Four Test Suite Reduction Techniques,” Proceedings of the 28th International Conference on Software Engineering, Shanghai, China, pp. 636–640, May 2006.
[47] G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong, “An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites,” Proceedings of the International Conference on Software Maintenance, Bethesda, MD, pp. 34–43, Nov. 1998.
[48] T. Y. Chen and M. F. Lau, “A New Heuristic for Test Suite Reduction,” Information & Software Technology, Vol. 40, Issues 5–6, pp. 347–354, Jul. 1998.
[49] B. Jiang, K. Zhai, W. K. Chan, T. H. Tse, and Z. Zhang, “On the Adoption of MC/DC and Control-Flow Adequacy for A Tight Integration of Program Testing and Statistical Fault Localization,” Information & Software Technology, Vol. 55, Issue 5, pp. 897–917, May 2013.
[50] W. Afzal, R. Torkar, and R. Feldt, “A Systematic Review of Search-Based Testing for Non-Functional System Properties,” Information & Software Technology, Vol. 51, Issue 6, pp. 957–976, Jun. 2009.
[51] J. von Ronne., Test Suite Minimization: An Empirical Investigation, University Honors College Thesis, Oregon State University, Corvallis, USA, 1999.
[52] P. C. Jorgensen, Software Testing: A Craftsman's Approach, 4th Edition, CRC Press, 2013.
[53] N. Fenton and J. Bieman, Software Metrics: A Rigorous and Practical Approach, 3rd Edition, CRC Press, 2015.
[54] W. E. Howden, “Reliability of the Path Analysis Testing Strategy,” IEEE Transactions on Software Engineering, Vol. SE–2, No. 3, Sept. 1976.
[55] J. C. Huang, “An Approach to Program Testing,” ACM Computing Surveys (CSUR), Vol. 7, Issue 3, pp. 113–128, Sept. 1975.
[56] H. Schiller, Using MEMIMAP to measure the extent of program testing, IBM Systems Development Division, Poughkeepsie, New York, Feb. 1969.
[57] S. Sampath, S. Sprenkle, E. Gibson, and L. Pollock, “Integrating Customized Test Requirements with Traditional Requirements in Web Application Testing,” Proceedings of the 2006 Workshop on Testing, Analysis, & Verification of Web Services & Applications (TAV–WEB), Portland, Maine, pp. 23–32, Jul. 2006.
[58] G. J. Myers, The Art of Software Testing, 2nd edition, Wiley, 2004.
[59] P. G. Frankl, S. N. Weiss, and E. J. Weyuker, “ASSET: A System to Select and Evaluate Tests,” Proceedings of IEEE Conference on Software Tools, New York, pp. 72–79, Apr. 1985.
[60] J. R. Horgan and S. London, “Data Flow Coverage and the C Language,” Proceedings of the Symposium on Testing, Analysis, and Verification, Victoria, British Columbia, Canada, pp. 87–97, Oct. 1991.
[61] J. Laski, “Data Flow Testing in STAD,” Journal of Systems and Software, Vol 12, Issue 1, pp 3–14, Apr. 1990.
[62] T. J. Ostrand and E. J. Weyuker, “Data Flow-Based Test Adequacy Analysis for Languages with Pointers,” Proceedings of the Symposium Testing, Analysis, and Verification, Victoria, British Columbia, Canada, pp 74–86, Oct. 1991.
[63] IEEE Standard 829-1998, IEEE Standard for Software Test Documentation, Dec. 1998.
[64] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithm, 3rd Edition, MIT Press, 2009.
[65] S. Tallam and N. Gupta, “A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization”, Proceedings of the 6th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE ’05), Lisbon, Portugal, pp. 35–42, Sept. 2005.
[66] S. Sprenkle, E. Sampath, A. Gibson, L. Pollock, and A. Souter, “An Empirical Comparison of Test Suite Reduction Techniques for User-Session-Based Testing of Web Applications,” Proceedings of the 21th International Conference on Software Maintenance (ICSM ‘05), Budapest, Hungary, pp. 549–558, Sept. 2005.
[67] C. Y. Huang, M. R. Lyu, and S. Y. Kuo, “A Unified Scheme of Some Nonhomogenous Poisson Process Models for Software Reliability Estimation,” IEEE Transactions on Software Engineering, Vol. 29, No. 3, pp. 261–269, Mar. 2003.
[68] P. S. Bullen., Handbook of Means and Their Inequalities, 2nd edition, Kluwer Academic Publishers, New York, 2003.
[69] "Arithmetic mean," in Wikipedia, Wikimedia Foundation, 2016. [Online]. Available: https://en.wikipedia.org/wiki/Arithmetic_mean. Accessed: Jul. 13, 2016.
[70] "Geometric mean," in Wikipedia, Wikimedia Foundation, 2016. [Online]. Available: https://en.wikipedia.org/wiki/Geometric_mean. Accessed: Jul. 13, 2016.
[71] "Harmonic mean," in Wikipedia, Wikimedia Foundation, 2016. [Online]. Available: https://en.wikipedia.org/wiki/Harmonic_mean. Accessed: Jul. 13, 2016.
[72] "Contraharmonic mean," in Wikipedia, Wikimedia Foundation, 2016. [Online]. Available: https://en.wikipedia.org/wiki/Contraharmonic_mean. Accessed: Jul. 13, 2016.
[73] "Cubic mean," in Wikipedia, Wikimedia Foundation, 2015. [Online]. Available: https://en.wikipedia.org/wiki/Cubic_mean. Accessed: Jul. 13, 2016.
[74] J. A. Jones and M. J. Harrold, “Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage,” IEEE Transactions on Software Engineering, Vol 29, No. 3, pp 195–209, Mar. 2003.
[75] J. Clarke, J. J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin, B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd, “Reformulating Software Engineering as a Search Problem,” IEE Proceedings - Software, Vol. 150, pp. 161–175, Jun. 2003.
[76] http://sir.unl.edu/portal/, Software-artifact Infrastructure Repository, 2008. [Accessed Jun. 20, 2015].
[77] G. Keller and B. Warrack, Statistics for Management and Economics, Thomson, 2003.
[78] A. Arcuri and L. Briand, “A practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering,” Proceedings of the 33rd International Conference on Software Engineering (ICSE), Honolulu, HI, pp. 1–10, May 2011.
[79] S. Kpodjedo, F. Ricca, G. Antoniol and P. Galinier, “Evolution and Search Based Metrics to Improve Defects Prediction,” Proceedings of the 1st International Symposium on Search Based Software Engineering, Windsor, pp. 23–32, May 2009.
[80] G. Keller, Managerial Statistics, 9th Edition, 2012.
[81] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering, Berlin; New York: Springer, 2012.
[82] M. Pezze and M. Young, Software Testing and Analysis: Process, Principles and Techniques, John Wiley & Sons, Inc, New York, NY, USA, 2007.
[83] T. Y. Chen, and M. F. Lau, “Dividing Strategies for the Optimization of a Test Suite,” Information Processing Letters, Vol. 60, Issue 3, pp. 135–141, Nov. 1996.