Classification of open-ended responses to a research-based assessment using natural language processing
Open Access
- 2 June 2022
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review Physics Education Research
- Vol. 18 (1), 010141
- https://doi.org/10.1103/physrevphyseducres.18.010141
Abstract
Surveys have long been used in physics education research to understand student reasoning and inform course improvements. However, to make analysis of large sets of responses practical, most surveys use a closed-response format with a small set of potential responses. Open-ended formats, such as written free response, can provide deeper insights into student thinking, but take much longer to analyze, especially with a large number of responses. Here, we explore natural language processing as a computational solution to this problem. We create a machine learning model that can take student responses from the Physics Measurement Questionnaire as input, and output a categorization of student reasoning based on different reasoning paradigms. Our model yields classifications with the same level of agreement as that between two humans categorizing the data, but can be done by a computer, and thus can be scaled for large datasets. In this work, we describe the algorithms and methodologies used to create, train, and test our natural language processing system. We also present the results of the analysis and discuss the utility of these approaches for analyzing open-response data in education research.Funding Information
- National Science Foundation (PHY-1734006)
- Norges Forskningsråd (288125)
- Olav Thon Stiftelsen
- Direktoratet for internasjonalisering og kvalitetsutvikling i høgare utdanning
- Lappan-Philips Foundation
This publication has 28 references indexed in Scilit:
- Resource Letter RBAI-1: Research-Based Assessment Instruments in Physics and AstronomyAmerican Journal of Physics, 2017
- A Learning Analytics Methodology for Detecting Sentiment in Student Fora: A Case Study in Distance EducationEuropean Journal of Open, Distance and E-Learning, 2015
- Human vs. Computer Diagnosis of Students’ Natural Selection Knowledge: Testing the Efficacy of Text Analytic SoftwareJournal of Science Education and Technology, 2011
- The future of natural selection knowledge measurement: A reply to Anderson et al. (2010)Journal of Research in Science Teaching, 2009
- Measuring knowledge of natural selection: A comparison of the CINS, an open‐response instrument, and an oral interviewJournal of Research in Science Teaching, 2008
- Impact of a conventional introductory laboratory course on the understanding of measurementPhysical Review Special Topics - Physics Education Research, 2008
- The development of first year physics students' ideas about measurement in terms of point and set paradigmsInternational Journal of Science Education, 2001
- Point and set reasoning in practical science measurement by entering university freshmenScience Education, 2001
- First‐year physics students’ perceptions of the quality of experimental measurementsInternational Journal of Science Education, 1998
- The automatic identification of stop wordsJournal of Information Science, 1992