Large Language Model for Assignment Feedback on Open-ended Subjective Questions
Abstract
Feedback is a key factor in motivating and consolidating learning, but in classroom teaching, a teacher needs to provide timely and effective feedback on the homework of dozens of students, which puts much pressure on the teacher. Meanwhile, existing automatic feedback systems are not suitable for open writing tasks. The emergence of ChatGPT has attracted the attention of researchers. We selected 28 open-ended subjective question assignments to input into ChatGPT, and compared and analyzed the scores and comments generated by ChatGPT with those of teachers, to explore the feasibility of using a Large Language Model to provide timely feedback for open-ended subjective questions. Our research indicates that ChatGPT can score and evaluate learners' open-ended subjective homework, and the rating of ChatGPT can be similar to that of teachers. Moreover, ChatGPT's comments can be similar in terms of focus and emotional bias to those of teachers, indicating that ChatGPT's ratings and comments have high credibility.
References
Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70-120. https://doi.org/10.3102/0034654312474350
Higgins, R., Hartley, P., & Skelton, A. (2002). The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64. https://doi.org/10.1080/03075070120099368
Jia, Q., Young, M., Xiao, Y., Cui, J., Liu, C., Rashid, P., & Gehringer, E. (2022). Insta-Reviewer: A Data-Driven Approach for Generating Instant Feedback on Students' Project Reports. International Educational Data Mining Society.
Keuning, H., Jeuring, J., & Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE), 19(1), 1-43. https://doi.org/10.1145/3231711
Kim, D., Kwon, Y., Liu, P., Kim, I. L., Perry, D. M., Zhang, X., & Rodriguez-Rivera, G. (2016). Apex: automatic programming assignment error explanation. ACM SIGPLAN Notices, 51(10), 311-327. https://doi.org/10.1145/3022671.2984031
Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2), 211. https://doi.org/10.1037/0033-295X.104.2.211
Lewis, M. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. https://doi.org/10.18653/v1/2020.acl-main.703
Li, C., & Xing, W. (2021). Natural language generation using deep learning to support MOOC learners. International Journal of Artificial Intelligence in Education, 31, 186-214. https://doi.org/10.1007/s40593-020-00235-x
Marwan, S., Gao, G., Fisk, S., Price, T. W., & Barnes, T. (2020, August). Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In Proceedings of the 2020 ACM Conference on International Computing Education Research (pp. 194-203). https://doi.org/10.1145/3372782.3406264
Nagata, R., Vilenius, M., & Whittaker, E. (2014, June). Correcting preposition errors in learner English using error case frames and feedback messages. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 754-764). https://doi.org/10.3115/v1/P14-1071
Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G., & Melis, E. (2014). Exploring feedback and student characteristics relevant for personalizing feedback strategies. Computers & Education, 71, 56-76. https://doi.org/10.1016/j.compedu.2013.09.011
Pardo, A., Bartimote, K., Shum, S. B., Dawson, S., Gao, J., Gašević, D., ... & Vigentini, L. (2018). Ontask: Delivering data-informed, personalized learning support actions. Journal of Learning Analytics, 5(3), 235-249. https://doi.org/10.18608/jla.2018.53.15
Price, T., Zhi, R., & Barnes, T. (2017). Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming. International Educational Data Mining Society.
Radford, A. (2018). Improving language understanding by generative pre-training.
Wu, L. J., & Chang, K. E. (2023). Effect of embedding a cognitive diagnosis into the adaptive dynamic assessment of spatial geometry learning. Interactive Learning Environments, 31(2), 890-907. https://doi.org/10.1080/10494820.2020.1815216
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).
1.png)
