Badges in education are an increasingly popular phenomenon, and given the relative recency of this phenomenon, a variety of questions exists as to the abilities and effectiveness of
badges. In this study, the effect of digital badges within a Moodle-based online homework system was studied for an undergraduate general physics course at a large research-based university in northeast Taiwan. Students (N=162) self-selected themselves into two course sections; students in one course section (N = 68) were able to earn one badge per assignment for turning their assignments in earlier than the assignment deadline – treatment group, while students in the other section (N = 94) could not – control group. In addition to submission before a special badge deadline, students in the treatment group were also required to obtain
maximum scores to get these badges. However, assignments were designed to be easy enough for students to earn the maximum grade (which students generally did). Additionally,
students in the treatment group were able to earn higher-level badges by combining the assignment badges. Students in the treatment group were intended to be fully aware of
badges, and badge requirements; moreover, the badge design was visible.

The study results showed that students in the treatment group turned in their assignments earlier than students in the control group did, and this difference was statistically significant. However, the fact that students in both groups came from markedly different departments weakened the internal validity of the study’s results. Further analysis showed that students in the treatment group spaced their assignment practice more than students in the control group
did, and the difference was statistically significant. Additionally, students in the treatment group actively attempted to earn badges, as there was a statistically significant increase in the number of badges earned by students in the treatment group over those in the control group. Based on a questionnaire given to study participants towards the conclusion of the study, the study found that students’ perception of badges was positive. All of the aforementioned gains were observed despite an absence of any statistically significant differences in the learning
outcomes of both study groups.

These findings corroborate earlier findings by other researchers that badges can be used to motivate specific behaviours in students whilst requiring minimal changes to the course structure. However, further corroborating earlier research is the finding that badges may not be particularly useful to motivate students towards challenging tasks. An earlier study of this course, in a preceding academic year found that students are appreciative of the online homework system; and it appears from this study that the primary function of badges within the system is to enhance the experience of students, as well as to motivate timely engagement with assignments.

Badges in education are an increasingly popular phenomenon, and given the relative recency of this phenomenon, a variety of questions exists as to the abilities and effectiveness of
badges. In this study, the effect of digital badges within a Moodle-based online homework system was studied for an undergraduate general physics course at a large research-based university in northeast Taiwan. Students (N=162) self-selected themselves into two course sections; students in one course section (N = 68) were able to earn one badge per assignment for turning their assignments in earlier than the assignment deadline – treatment group, while students in the other section (N = 94) could not – control group. In addition to submission before a special badge deadline, students in the treatment group were also required to obtain
maximum scores to get these badges. However, assignments were designed to be easy enough for students to earn the maximum grade (which students generally did). Additionally,
students in the treatment group were able to earn higher-level badges by combining the assignment badges. Students in the treatment group were intended to be fully aware of
badges, and badge requirements; moreover, the badge design was visible.

The study results showed that students in the treatment group turned in their assignments earlier than students in the control group did, and this difference was statistically significant. However, the fact that students in both groups came from markedly different departments weakened the internal validity of the study’s results. Further analysis showed that students in the treatment group spaced their assignment practice more than students in the control group
did, and the difference was statistically significant. Additionally, students in the treatment group actively attempted to earn badges, as there was a statistically significant increase in the number of badges earned by students in the treatment group over those in the control group. Based on a questionnaire given to study participants towards the conclusion of the study, the study found that students’ perception of badges was positive. All of the aforementioned gains were observed despite an absence of any statistically significant differences in the learning
outcomes of both study groups.

These findings corroborate earlier findings by other researchers that badges can be used to motivate specific behaviours in students whilst requiring minimal changes to the course structure. However, further corroborating earlier research is the finding that badges may not be particularly useful to motivate students towards challenging tasks. An earlier study of this course, in a preceding academic year found that students are appreciative of the online homework system; and it appears from this study that the primary function of badges within the system is to enhance the experience of students, as well as to motivate timely engagement with assignments.

Abstract I
Acknowledgements II
Table of Contents III
List of Figures VI
List of Tables VIII
CHAPTER 1 Introduction 1
1.1 Background – An Introduction to Digital Badges in Education 2
1.2 Rationale for Study 3
1.3 Context of the Study 4
1.4 Research Questions 5
1.5 Limitations of the Study 7
1.6 Glossary of Terms 7
1.7 Summary of chapters 7
CHAPTER 2 Literature Review 9
2.1 The Teaching of Introductory Physics 9
2.1.1 A Review of Physics Education with Higher Education Institutions 9
2.1.2 Teaching of Introductory Physics with Web-based Homework System 12
2.2 Graduate Attributes 16
2.2.1 The Evolution of the Concept 16
2.2.2 Definition and Selection of Competencies: Theoretical and Conceptual Foundations (DeSeCo) 17
2.2.3 Towards a Framework for Graduate Attributes 20
2.3 Digital Badges 23
2.3.1 The Origins of Digital Badges 23
2.3.2 Revisiting the Utility of Badges – Badges to Develop Lifelong Learning Skills 24
2.3.3 The Perception and Potential of Digital Badges in Formal Higher Education 25
2.3.4 The Social Psychology of Badges 26
2.3.5 Quantitative Studies on Badges in Education 28
2.4 Distributed Practice 34
2.4.1 Experimental Studies on Distributed Practice for complex learning tasks. 34
2.5 Summary of Literature Review 36
CHAPTER 3 Methods 38
3.1 Study Design 38
3.1.1 Overview 38
3.2 Sample 39
3.2.1 Setting and Sample Size 39
3.3 Experimental Procedure 41
3.3.1 Procedures prior to experimentation for both groups 41
3.3.2 Procedures following commencement of experimentation for both groups 42
3.4 Summary of Instruments 42
3.4.1 End of Semester Survey about Badges 43
3.5 Data Management 43
3.6 Data Analysis 44
3.6.1 Primary Outcomes 44
3.6.2 Secondary Outcomes 45
3.6.3 Additional Outcomes 45
3.7 Contamination 46
CHAPTER 4 System Design 47
4.1 Chapter Outline 47
4.2 Formative evaluation survey of badges 48
4.2.1 Sample 48
4.2.2 Survey Layout and Key Findings 49
4.3 Badge Design 54
4.3.1 Visual appearance 54
4.3.2 Requirements 56
4.4 Homework System 59
4.4.1 Technical Details 59
4.4.2 Assignment System 60
4.4.3 Badges 68
CHAPTER 5 Results 71
5.1 Chapter Outline 71
5.2 Analysis of Study Sample 73
5.3 Research Question: Timeliness 75
5.3.1 Analysis of Raw Timeliness Data 75
5.3.2 Analysis of Categorized Timeliness data 80
5.3.3 Panel Data Model 82
5.4 Research Question: Distributed Practice 86
5.4.1 Calculation of Distributed Practice 86
5.4.2 Analysis of Raw Inter-session Interval data 87
5.4.3 Hypothesis Testing: How is study group related to distributed practice? 87
5.5 Additional Outcomes 89
5.5.1 Research Question: Badges 89
5.5.2 Research Question: Students’ perception of badges 92
5.5.3 Research Question: Performance 100
CHAPTER 6 Discussion and Conclusion 104
6.1 Timeliness 104
6.1.1 Sample 104
6.1.2 Modelling 105
6.1.3 Summary 106
6.2 Distributed Practice 107
6.3 Additional Analysis 108
6.3.1 Badges 108
6.3.2 End of Semester Survey about Badges. 109
6.4 Conclusion 111
6.4.1 A summary of findings in relation to research questions 111
6.4.2 Overall Significance of the Study 112
6.5 Future research 113
References 115
APPENDIX A Visualization of Database Logs 120
APPENDIX B Survey Instruments 121
APPENDIX C System Artefacts 130

Abramovich, S., Schunn, C., & Higashi, R. (2013). Are badges useful in education?: it depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61, 217-232.
Adams, W. K., Perkins, K. K., Podolefsky, N. S., Dubson, M., Finkelstein, N. D., & Weiman, C. E. (2006). New instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey. Physical Review Special Topics-Physics Education Research, 2(1).
Allain, R., & Williams, T. (2006). The Effectiveness of Online Homework in an Introductory Science Class. Journal of College Science Teaching, 35(6), 28-30.
Antin, J., & Churchill, E. F. (2011). Badges in Social Media: A Social Psychological Perspective. CHI 2011 Gamification Workshop Proceedings. Vancouver: CHI.
Ariely, D., & Norton, M. (2009). Conceptual Consumption. Annual Review of Psychology, 60(1), 475-499.
Arora, M. L., Rho, Y. J., & Masson, C. (2013). Longitudinal Study of Online Statics Homework as a Method to Improve Learning. Journal of STEM Education : Innovations and Research, 14(1), 36-44.
Aspin, D., Chapman, J., Evans, K., & Bagnall, R. (2012). Introduction and Overview. In D. Aspin, J. Chapman, K. Evans, & R. Bagnall (Eds.), Second International Handbook of Lifelong Learning (pp. xlv-lxxxiv). Netherlands: Springer Netherlands.
Barrie, S. C. (2004, August 3). A research-based approach to generic graduate attributes policy. Higher Education Research & Development, 23(3), 261-275.
Barrie, S. C., Jain, P., & Carew, A. (2003). Generic Graduate Attributes: A Research Based Framework for a Shared Vision. Staff and Educational Development International, 7(3), 191-199.
Bloom, K. C., & Shuell, T. J. (1981). Effects of massed and distributed practice on the learning and retention of second-language vocabulary. The Journal of Educational Research, 74(4), 245-248.
Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003). Comparison of Student Performance Using Web andPaper-Based Homework in College-Level Physics. Journal of Research in Science Teaching, 40(10), 1050-1071. doi:10.1002/tea.10120
Brewe, E., Kramer, L., & O'Brien, G. (2009). Modeling instruction: Positive attitudinal shifts in introductory physics measured with CLASS. Physical Review Special Topics-Physics Education Research, 5(1).
Dancy, M., & Henderson, C. (2010). Pedagogical practices and instructional change of physics faculty. American Journal of Physics, 78(10), 1056-1063.
Dawes, R., Van De Kragt, A., & Orbell, J. (1988). Not me or thee but we: The importance of group identity in eliciting cooperation in dilemma situations: Experimental manipulations. Acta Psychologica, 68(1-3), 83-97.
Demirci, N. (2007). University Students' Perceptions of Web-based vs. Paper-based Homework in a General Physics Course. Eurasia Journal of Mathematics, Science & Technology Education, 3(1), 29-34.
Denny, P. (2013). The Effect of Virtual Achievements on Student Engagement. Proceedings of the SIGCHI conference on human factors in computing systems (pp. 763-772). Paris: ACM.
Donovan, J. J., & Radosevich, D. J. (1999, October). A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology, 84(5), 795-805.
Down, C. (2006). Lifelong Learning, Graduate Capabilities and Workplace Learning. In P. Hager, & Holland Susan (Eds.), Graduate Attributes, Learning and Employability (pp. 187-205). Netherlands: Springer.
Dufresne, R. J., Hart, D., Mestre, J. P., & Rath, K. (2002). The Effect of Web-Based Homework on Test Performance in Large Enrollment Introductory Physics Courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229-251.
Faure, E., Herrera, F., Kaddoura, A.-R., Lopes, H., Petrovsky, A. V., Rahnema, M., & Ward, F. C. (1972). Learning to be: The world of education today and tomorrow. UNESCO. Paris: UNESCO.
Fraser, J. M., Timan, A. L., Miller, K., Dowd, J. E., Tucker, L., & Mazur, E. (2014). Teaching and physics education research: bridging the gap. Reports on Progress in Physics, 77(3). doi:10.1088/0034-4885/77/3/032401
Gaffney, M. A., Ryan, D., & Wurst, C. (2010). Do Online homework systems improve student performance? In A. H. Catanach, & D. Feldmann, Advances in Accounting Education (pp. 49 - 68). Emerald Group Publishing Limited.
Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., & Knight, E. (2015). Digital badges in education. Education and Information Technologies, 20(2), 403-410. doi:10.1007/s10639-013-9291-7
Glenberg, A. M. (1977). Influences of retrieval processes on the spacing effect in free recall. Journal of Experimental Psychology: Human Learning and Memory, 3(3), 282-294.
Glenberg, A., & Lehmann, T. (1980). Spacing repetitions over 1 week. Memory & Cognition, 8(6), 528-538.
Glover, I., & Latif, F. (2013). Investigating perceptions and potential of open badges in formal higher education. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1398-1402). Victoria, Canada: AACE.
Gok, T. (2011). Comparison of student performance using web- and paper- based homework in large enrollment introductory physics courses. International Journal of the Physical Sciences, 6(15), 3778-3784. doi:10.5897/IJPS11.217
Gonczi, A. (2006). The OECD: Its Role in the Key Competences Debate and in the Promotion of Lifelong Learning. In P. Hager, & S. Holland (Eds.), Graduate Attributes, Learning and Employability (pp. 105-124). Netherlands: Springer.
Grote, M. (1995, June). The Effect of Massed Versus Spaced Practice on Retention and Problem-Solving in High School Physics. Ohio Journal of Science, 95(3), 243-247.
Haaranen, L., Ihantola, P., Hakulinen, L., & Korhonen, A. (2014). How (not) to Introduce Badges to Online Exercises. Proceedings of the 45th ACM technical symposium on Computer science education (pp. 33-38). New York: ACM.
Hager, P., & Holland, S. (2006). Introduction. In P. Hager, & S. Holland (Eds.), Graduate Attributes, Learning and Employability (pp. 1-48). Netherlands: Springer.
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(64). doi:10.1119/1.18809
Hakulinen, L., Auvinen, T., & Korhonen, A. (2013). Empirical Study on the Effect of Achievement Badges in TRAKLA2 Online Learning Environment. Learning and Teaching in Computing and Engineering (LaTiCE) (pp. 47-54). Macau: IEEE.
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, 152-161.
Harvey, L. (2000). New realities: The relationship between higher education and employment. Tertiary Education & Management, 6(1), 3-17.
Hestenes, D., & Wells, M. (1992). A Mechanics Baseline Test. The Physics Teacher, 30, 158-166.
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30, 141-158.
Hinchliffe, G. (2006). Graduate Employability and Lifelong Learning: A need for Realism? In Graduate Attributes, Learning and Employability (pp. 91-104). Netherlands: Springer.
Hintzman, D. (1974). Theoretical implications of the spacing effect. In R. Solso (Ed.), Theories in cognitive psychology: The Loyola Symposium (p. 386). Oxford, England: Lawrence Erlbaum.
Ho, D., Imai, K., King, G., & Stuart, E. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15, 199-236. Retrieved from http://j.mp/jPupwz
Hung, K. C. (2015). A Study of Integrating Homework System into University General Physics. Degree of Master of Science, National Tsing Hua University, Institute of Learning Sciences, Hsinchu, Taiwan.
Joseph, B. (2012, June 25). Six Ways to Look at Badging Systems Designed for Learning. Retrieved February 3, 2015, from Online Leadership Program: http://www.olpglobalkids.org/content/six-ways-look-badging-systems-designed-learning/
Joseph, B. (2012). The Introduction of Badges at the Epstein Jewish Day School. Global Kids. Global Kids.
Krug, D., Davis, B. T., & Glover, J. A. (1990). Massed versus distributed repeated reading: A case of forgetting helping recall? Journal of Educational Psychology, 82(2), 366-371.
Liang, J. (2002). Study of the Effectiveness of a Web-based Interactive Homework. Degree of Master of Science, Mississippi State University.
Linderoth, J. (2012, March 27). Why gamers don't learn more. An ecological approach to games as learning environments. Journal of Gaming & Virtual Worlds, 4(1), 45-62.
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., . . . Kraut, R. E. (2005). Using social psychology to motivate contributions to online communities. Journal of Computer-Mediated Communication, 10(4).
Lynch, R., & Dembo, M. (2004). The Relationship Between Self-Regulation and Online Learning in a Blended Learning Context. The International Review of Research in Open and Distributed Learning, 5(2).
Martín-Blas, T., & Serrano-Fernández, A. (2009). The role of new technologies in the learning process: Moodle as a teaching tool in Physics. Computers & Education, 52(1), 35-44. doi: 10.1016/j.compedu.2008.06.005
Melton, A. W. (1970, October). The situation with respect to the spacing of repetitions and memory. Journal of Verbal Learning and Verbal Behavior, 9(5), 596-606.
Montola, M., Nummenmaa, T., Lucero, A., Boberg, M., & Korhonen, H. (2009). Applying game achievement systems to enhance user experience in a photo sharing service. Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era on - MindTrek '09 (pp. 94-97). New York: ACM.
Morrison, B. B., & DiSalvo, B. (2014). Khan academy gamifies computer science. Proceedings of the 45th ACM technical symposium on Computer science education (pp. 39-44). New York: ACM. doi:https://dl.acm.org/citation.cfm?id=2538946
Mozilla Foundation. (2014). About | Open Badges. Retrieved February 3, 2015, from Open Badges: http://openbadges.org/about/
OECD. (1996). Lifelong learning for all. OECD. Paris: OECD.
OECD. (2001). Definition and Selection of Competencies: Theoretical and Conceptual Foundations (DeSeCo). OECD. OECD.
Rohrer, D., & Taylor, K. (2006, July 6). The Effects of Overlearning and Distributed Practice on the Retention of Mathematics Knowledge. Applied Cognitive Psychology, 20(9), 1209-1224. doi:10.1002/acp.1266
Rychen, D., & Salganik, L. (2000). Definition and Selection of Key Competencies. The INES Compendium: Contributions from the INES Networks and Working Groups (pp. 61-73). Tokyo: INES, OECD.
Thoennessen, M., & Harrison, M. J. (1996). Computer-assisted assignments in a large physics class. Computers & Education, 27(2), 141-147.
Turpen, C., & Finkelstein, N. D. (2009). Not all interactive engagement is the same: variations in physics professors' implementation of Peer Instruction. Physical Review Special Topics-Physics Education Research, 5(2).
Turpen, C., & Finkelstein, N. D. (2010). The construction of different classroom norms during Peer Instruction: Students perceive differences. Physical Review Special Topics-Physics Education Research, 6(2).
Wells, M., Hestenes, D., & Swackhamer, G. (1995). A modeling method for high school physics instruction. American Journal of Physics, 63(7), 606-619.
Zimmerman, B. J. (2002). Becoming a Self-Regulated Learner: An Overview. Theory Into Practice, 41(2), 64-70.