軟體問題是現今系統故障的主要原因，並且會導致財務損失，而軟體安全性弱點便是其中最具威脅性的一種。事實上，軟體安全性弱點是一種特殊的軟體錯誤。軟體工程師常常無法持續追蹤其他開發者所報告及修復的弱點。量化品質標準常常屈服於人力、時間、資源的不足。軟體安全性弱點一般難以量化，並且更難以預測或涉及任何流程改進的計劃和活動。有許多關於如何偵測或避免軟體安全性弱點的研究，而這篇研究我們則是專注於軟體安全性弱點發現數量的預測，這可以幫助我們改善應用軟體的安全部署。文獻上已經有一些弱點發現模型被提出，但這些模型需要目標應用程式的大量弱點資料以正常運作，並且無法適應於弱點發現趨勢在短時間內急遽增加或減少的現象，因此，我們提出了帕爾曲線變動點模型以及一個初步預測方案用以預測軟體中的弱點。一開始應用程式中的總弱點數會先被概略地預測，在此之後當有足夠的軟體歷史弱點資料時，就可以進一步地使用我們的帕爾曲線變動點模型來預測。我們提出的這個弱點發現模型能夠藉由將變動點插入到資料中適當的位置以適應短時間內急遽增加或減少的弱點發現趨勢。在實驗中，我們使用了帕爾曲線變動點模型以及初步預測方案預測了Mozilla Firefox這個軟體的弱點發現量，結果顯示，我們提出的帕爾曲線變動點模型確實勝過了其他弱點發現模型，特別是在那些改變幅度很大的弱點資料上更為明顯。

Software problems are the main cause of system failures today which can lead to financial losses and software vulnerabilities as the greatest threats. Security vulnerabilities are a particular case of software faults. Software engineers are often unable to keep track of vulnerabilities that other developers have reported and solved. Quantitative quality metrics are often subdued due to lack of adequate amounts of people, time, and resources. Security-related software vulnerabilities are typically hard to quantify and even harder to predict or relate to any process improvement initiatives and activities. There is much research about how to detect or avoid software vulnerabilities; however, in this work we focus on software vulnerability discovery prediction, which can help improve the secure deployment of software applications. There are a few vulnerability discovery models (VDMs) proposed in literature, yet those models require a high quantity of vulnerability data of the target application in order to work; further, VDMs cannot adapt to abrupt increment or decrement of the vulnerability discovery trends in a short time interval. Hence, a generalized Parr-curve (GPC) model with an initial prediction scheme is proposed and used in order to predict software vulnerabilities. The number of total vulnerabilities in the application is first roughly predicted; afterward, the GPC model can be further used when the software historical vulnerability data is available. The proposed vulnerability discovery model can adapt to abrupt increment or decrement of the discovery trends in a short time interval by applying change-points to proper positions of the data. In the experiment, we have used the GPC model with the initial prediction scheme in order to predict vulnerability discovery in Mozilla Firefox. The result shows that the proposed GPC model outperforms other VDMs, especially in those with abruptly changing data.

Abstract in Chinese i
Abstract ii
Acknowledgements iv
Contents v
List of Tables vi
List of Figures vii
Notations viii
Chapter 1 Introduction 1
Chapter 2 Related Works 8
2.1 Review of Vulnerability Discovery Models and Resource Models 9
2.2 Change-Points and Vulnerability Discovery Data 16
Chapter 3 Methods of Combinatorial Model 21
3.1 The Factors Causing Defects in the Software 21
3.1.1 Phase Factor (F_ph) 22
3.1.2 Programming Team Factor (F_pt) 23
3.1.3 Process Maturity Factor (F_m) 24
3.1.4 Software Structure Factor (F_s) 26
3.2 Generalized Parr-Curve Model 28
Chapter 4 Experiments and Data Analysis 32
4.1 Data Sets 32
4.2 Parameter Estimation 35
4.3 Evaluation Criteria 36
4.4 Performance Analysis 41
4.4.1 Preliminary Prediction of Using the Combinatorial Model 42
4.4.2 Performance Comparison and Assessment 43
4.5 Threats to Validity 49
Chapter 5 Conclusions 51
Appendixes 53
Appendix A Original Vulnerability Data of Mozilla Firefox 3.6 53
Appendix B Reorganized Vulnerability Data of Mozilla Firefox 3.6 85
References 88

[1] Gateways to Infection: Exploiting Software Vulnerabilities [Online]. Available: http://about-threats.trendmicro.com/RelatedThreats.aspx?language=tw&name=Gateways+to+Infection%3A+Exploiting+Software+Vulnerabilities
[2] M. Dowd, J. McDonald, and J. Schuh, The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities, Addison-Wesley Professional 2006.
[3] A. Cencini, K. Yu, and T. Chan, Software Vulnerabilities: Full-, Responsible-, and Non-Disclosure, University of Washington, Dec. 2005.
[4] M. McKeay. (2014, Oct. 16). Heartbleed and Shellshock: The New Norm in Vulnerabilities [Online]. Available: https://securityintelligence.com/heartbleed-and-shellshock-the-new-norm-in-vulnerabilities/
[5] A. Greenberg. (2014, Dec. 29). The 5 Most Dangerous Software Bugs of 2014 [Online]. Available: https://www.wired.com/2014/12/most-dangerous-software-bugs-2014/
[6] P. Mutton. (2014, Apr. 8). Half a million widely trusted websites vulnerable to Heartbleed bug [Online]. Available: http://news.netcraft.com/archives/2014/04/08/half-a-million-widely-trusted-websites-vulnerable-to-heartbleed-bug.html
[7] A. Greenberg. (2014, Sep. 25). Hackers Are Already Using the Shellshock Bug to Launch Botnet Attacks [Online]. Available: https://www.wired.com/2014/09/hackers-already-using-shellshock-bug-create-botnets-ddos-attacks/
[8] J.Saarinen. (2014, Sep. 26). First Shellshock botnet attacks Akamai, US DoD networks [Online]. Available: http://www.itnews.com.au/news/first-shellshock-botnet-attacks-akamai-us-dod-networks-396197
[9] N. Perlroth. (2014, Sep. 26). Companies Rush to Fix Shellshock Software Bug as Hackers Launch Thousands of Attacks [Online]. Available: http://bits.blogs.nytimes.com/2014/09/26/companies-rush-to-fix-shellshock-software-bug-as-hackers-launch-thousands-of-attacks/
[10] National Institute of Standards and Technology [Online]. Available: http://www.nist.gov/
[11] J. H. Allen et al., Software Security Engineering: A Guide for Project Managers. Addison-Wesley Professional, 2008.
[12] E. Rescorla, “Is finding security holes a good idea?,” IEEE Security & Privacy, Vol. 3, No. 1, pp. 14–19, Jan./Feb. 2005.
[13] O. Alhazmi and Y. Malaiya, “Prediction capabilities of vulnerability discovery models,” Proceedings of RAMS ’06. Annual Reliability Maintainability Symposium, pp. 86–91, Jan. 2006.
[14] V. H. Nguyen and F. Massacci, “An Independent Validation of Vulnerability Discovery Models,” Engineering Secure Software and Systems, Vol. 7159 of the series Lecture Notes in Computer Science, pp. 89-96, 2012, DOI: 10.1007/978-3-642-28166-2_9.
[15] H. Okhravi and D. M. Nicol, “Evaluation of patch management strategies,” International Journal of Computational Intelligence : Theory and Practice, Vol. 3, No. 2, pp. 103-111, Dec. 2008.
[16] National Vulnerability Database [Online]. Available: http://nvd.nist.gov/
[17] The Open Source Vulnerability Database [Online]. Available: http://osvdb.org/
[18] O. H. Alhazmi and Y. K. Malaiya, “Application of vulnerability discovery models to major operating systems,” IEEE Transactions on Reliability, Vol. 57, No. 1, pp. 14–22, Mar. 2008.
[19] F. Parr, “An Alternative to the Rayleigh Curve Model for Software Development Effort,” IEEE Transactions on Software Engineering, Vol. SE-6, No. 3, pp. 291-296, May 1980.
[20] S. Z. Ke and C. Y. Huang, “Measurement and Analysis of Software Reliability Model with Parr-Curve Testing-Effort Distribution and Change-Points,” Master’s thesis, Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan, 2012.
[21] Y. K. Malaiya and J. Denton, “What do the software reliability growth model parameters represent,” Proceedings of IEEE International Symposium on Software Reliability Engineering, Nov. 1997.
[22] B. Ray, D. Posnett, V. Filkov, and P. Devanbu, “A large scale study of programming languages and code quality in github,” Proceedings of the 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, pp. 155–165, Nov. 2014.
[23] M. R. Lyu, Handbook of Software Reliability Engineering, IEEE computer society press, 1996.
[24] O. H. Alhazmi, and Y. K. Malaiya, “Quantitative Vulnerability Assessment of Systems Software,” Proceedings of 2005 Annual Reliability and Maintainability Symposium (RAMS'05), Jan. 2005.
[25] O. H. Alhazmi, and Y. K. Malaiya, “Modeling the Vulnerability Discovery Process,” Proceedings of 16th International Symposium on Software Reliability Engineering (ISSRE 2005), Nov. 2005.
[26] R. Anderson, “Security in Open Versus Closed Systems - the Dance of Boltzmann,” Coase and Moore Conference on Open Source Software Economics, Jun. 2002.
[27] O. H. Alhazmi, and Y. K. Malaiya, “Measuring and Enhancing Prediction Capabilities of Vulnerability Discovery Models for Apache and IIS HTTP Servers,” Proceedings of 17th International Symposium on Software Reliability Engineering (ISSRE 2006), Nov. 2006.
[28] O. H. Alhazmi, Y. K. Malaiya, and I. Ray, “Measuring, Analyzing and Predicting Security Vulnerabilities in Software Systems,” Computers & Security, Vol. 26, No. 3, pp. 219-228, 2007.
[29] F. Massacci, and V. H. Nguyen, “An Empirical Methodology to Evaluate Vulnerability Discovery Models,” IEEE Transactions on Software Engineering, Vol. 40, Issue 12, pp. 1147-1162, Sep. 2014.
[30] Y. M. Suvorova, M. A. Korotkova, and E. V. Korotkov, “Study of the Paired Change Points in Bacterial Genes,” IEEE/ACM Transactions on Computer Biology and Bioinformatics, Vol. 11, No. 5, pp. 955-964, Dec. 2014.
[31] C. T. Lin and C. Y. Huang, “Enhancing and Measuring the Predictive Capabilities of the Testing-Effort Dependent Software Reliability Models,” Journal of Systems and Software, Vol. 81, Issue 6, pp. 1025-1038, Jun. 2008.
[32] P. K. Kapur, H. Pham, A. Gupta, P. C. Jha, Software Reliability Assessment with OR Applications, Springer, 2011.
[33] G. Comert, and A. Bezuglov, “An Online Change-Point-Based Model for Traffic Parameter Prediction,” IEEE Transactions on Intelligent Transportation Systems, Vol. 14, Issue 3, pp. 1360-1369, Sep. 2013.
[34] H. C. Joh, J. Kim, and Y. K. Malaiya, “Vulnerability Discovery Modeling Using Weibull Distribution,” Proceedings of 19th International Symposium on Software Reliability Engineering (ISSRE 2008), pp. 299-300, Nov. 2008.
[35] O. H. Alhazmi,, Y. K. Malaiya, and I. Ray, “Security Vulnerabilities in Software Systems: A Quantitative Perspective,” Data and Applications Security XIX, pp. 281-294, Aug. 2005.
[36] J. Kim, “Vulnerability Discovery in Multiple Version Software Systems: A Open Source and Commercial Software System,” Master’s thesis, Department of Computer Science, Colorado State University, Fort Collins, CO, USA, 2007.
[37] P. L. Li, , M. Shaw, J. Herbsleb, B. Ray, and P. Santhanam, “Empirical Evaluation Of Defect Projection Models For Widely-Deployed Production Software Systems,” Proceedings of the 12th ACM SIGSOFT Twelfth International Symposium on Foundations Of Software Engineering (SIGSOFT ’04/FSE-12), Vol. 29, No. 6, pp. 263–272, Oct. 2004.
[38] J. D. Musa, A. Iannino, and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill, 1987.
[39] A. J. Perlis, F. Sayward, M. Shaw, Software Metrics: An Analysis and Evaluation, MIT Press, 1981.
[40] B. W. Boehm, Software Engineering Economics, Englewood Cliffs, Prentice-Hall, 1981.
[41] N. Ahmad, M. G. M. Khan, and L.S. Rafi, “A Study of Testing-Effort Dependent Inflection S-Shaped Software Reliability Growth Models with Imperfect Debugging,” International Journal of Quality & Reliability Management, Vol. 27, No. 1, pp. 89-110, 2010.
[42] R. S. Pressman, Software Engineering: A Practitioner's Approach, 8th Edition, McGraw-Hill, 2014.
[43] S. Chatterjee, R. B. Misra, and S. S. Alam, “Joint Effect of Test Effort and Learning Factor on Software Reliability and Optimal Release Policy,” International Journal of Systems Science, Vol. 28, Issue 4, pp. 391-396, 1997.
[44] K. Pillai and V. S. Sukumaran Nair, “A Model for Software Development Effort and Cost Estimation,” IEEE Transactions on Software Engineering, Vol. 23, No. 8, pp. 485-497, Aug. 1997.
[45] Firefox [Online]. Available: https://en.wikipedia.org/wiki/Firefox
[46] Linux Kernel [Online]. Available: https://en.wikipedia.org/wiki/Linux_kernel
[47] Y. P. Chang, “Estimation of Parameters for Nonhomogeneous Poisson Process: Software Reliability with Change-Point Model,” Communications in Statistics–Simulation and Computation, Vol. 30, Issue 3, pp. 623-635, 2001.
[48] H. J. Shyur, “A Stochastic Software Reliability Model with Imperfect-Debugging and Change-Point,” Journal of Systems and Software, Vol. 66, Issue 2, pp. 135-141, May 2003.
[49] M. Zhao, “Change-Point Problems in Software and Hardware Reliability,” Communications in Statistics - Theory and Methods, Vol. 22, Issue 3, pp. 757-768, 1993.
[50] F. Z. Zou, “A Change-Point Perspective on The Software Failure Process,” Software Testing, Verification and Reliability, Vol.13, Issue 2, pp. 85-93, Apr./Jun. 2003.
[51] S. J. Bae, T. Yuan, S. Ning, and W. Kuo, “A Bayesian Approach to Modeling Two-Phase Degradation Using Change-Point Regression,” Reliability Engineering and System Safety, Vol. 134, pp. 66-74, Feb. 2015.
[52] K. C. Chiu, Y. S. Huang, and T. Z. Lee, “A Study of Software Reliability Growth from the Perspective of Learning Effects,” Reliability Engineering and System Safety, Vol. 93, Issue 10, pp. 1410-1421, Oct. 2008.
[53] P. K. Kapur, D. N. Goswami, A. Bardhan, and O. Singh, “Flexible Software Reliability Growth Model with Testing Effort Dependent Learning Process,” Applied Mathematical Modelling, Vol. 32, Issue 7, pp. 1298-1307, Jul. 2008.
[54] X. Li, M. Xie, and S. H. Ng, “Sensitivity Analysis of Release Time of Software Reliability Models Incorporating Testing Effort with Multiple Change-Points,” Applied Mathematical Modelling, Vol. 34, Issue 11, pp. 3560-3570, Nov. 2010.
[55] Ian Sommerville, Software Engineering, Addison Wesley, 8th Edition, Addison-Wesley, 2007.
[56] D. Musa, A. Iannino, K. Okumoto, Software Reliability - Measurement, Prediction, Applications, McGraw-Hill, 1987.
[57] P. Piwowarski. M. Ohba and J. Caruso, “Coverage measurement Experience during Function Test,” Proceedings of the International Conference on Software Engineering, pp. 287-301, May 1993.
[58] M. Takahashi and Y. Kamayachi, “An Empirical Study of a Model for Program Error Prediction, in Software Reliability Models,” IEEE Computer Society, pp. 71-77, 1991.
[59] Paulk, Mark C. et al. (February 1993). “Capability Maturity Model for Software (Version 1.1),” Technical Report (Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University). CMU/SEI-93-TR-024 ESC-TR-93-177.
[60] N. Fenton & J. Bieman, Software Metrics: A Rigorous and Practical Approach, 3rd Edition, CRC Press, 2015.
[61] T. DeMarco, Controlling Software Projects: Management, Measurement, and Estimates, Prentice Hall, 1986.
[62] M. Jorgensen and M. Shepperd, “A Systematic Review of Software Development Cost Estimation Studies,” IEEE Transactions on Software Engineering, Vol. 33, No. 1, pp. 33-53, Jan. 2007.
[63] Tait, P., I. Vessey, “The effect of user involvement on system success: A contingency approach,” MIS Quarterly, pp. 91–108. Mar. 1988.
[64] M. Ohba, “Software reliability analysis models”, IBM Journal of research and Development, Vol. 28, Issue 4, pp. 428-443, Jul. 1984.
[65] Conte SD, Dunsmore HE, Shen V Y. Software Engineering Metrics and Models. Benjamin-Cummings Publishing Co. Inc: Redwood City, CA, 1986.
[66] S. P. Luan and C. Y. Huang, “An Improved Pareto Distribution for Modelling the Fault Data of Open Source Software,” Software Testing, Verification and Reliability, Vol. 24, Issue 6, pp. 416–437, Sep. 2014.