A Comparison of Rhetorical Move Analysis by GPT-4 and Humans in Abstracts of Scopus-Indexed Tourism Research Articles
Keywords:
ChatGPT, rhetorical moves, abstracts, tourism research articles, Scopus-indexed journal
Abstract
AI advancements have made ChatGPT a remarkable and versatile tool in education and linguistics, showcasing its potential to mimic human conversation and comprehend language. Scholars are intrigued by ChatGPT’s text data handling, yet its application in rhetorical move analysis remains largely unexplored. Therefore, the objective of this study is to investigate the ability of GPT-4 in the identification of rhetorical moves employed in the abstracts of tourism research articles indexed in Scopus. The essentiality of moves was also reported. Additionally, this research seeks to compare the accuracy of GPT-4’s analysis with that of humans. Adopting Hyland’s (2000) five-move model, the results indicated that GPT-4 analyzes moves more quickly but less accurately than human experts, and the four principal types of errors committed by GPT-4 include redundancy/over-count, unmatched categorization, incorrect sequence, and vague identification. The findings also revealed that Move 2 (Purpose) and Move 4 (Findings) are obligatory with a 100% essentiality rate through both GPT-4 and human analysis. Differences arise in certain steps of Move 1 (Introduction), Move 3 (Methods), and Move 5 (Conclusion), where GPT-4 often sees higher essentiality rates. This study shed light on the testament to AI’s current capabilities in move analysis in academic discourse.
References
Adeshola, I., & Adepoju, A. P. (2023). The opportunities and challenges of ChatGPT in education. Interactive Learning Environments, 1-14. https://doi.org/10.1080/10494820.2023.2253858
Alyousef, H. S. (2023). A multidimensional analysis of linguistic realizations and rhetorical move structure in geography research article abstracts: A corpus-based study. Atlantis: Revista de la Asociación Española de Estudios Anglo-Norteamericanos, 45(1), 190-212. https://doi.org/10.28914/Atlantis-2023-45.1.11
Bhatia, V. K. (1993). Analyzing Genre: Language Use in Professional Settings. Longman Publishing.
Biber, D. (2007). Discourse on the move: Using corpus analysis to describe discourse structure (Vol. 28). John Benjamins Publishing. https://doi.org/10.1075/scl.28
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. Preprints. https://doi.org/10.48550/arXiv.2303.12712
Can, S., Karabacak, E., & Qin, J. (2016). Structure of moves in research article abstracts in applied linguistics. Publications, 4(3), 1-16. https://doi.org/10.3390/publications4030023
Ding, H. (2007). Genre analysis of personal statements: Analysis of moves in application essays to medical and dental schools. English for specific purposes, 26(3), 368-392. https://doi.org/10.1016/j.esp.2006.09.004
Fauzan, U., Lubis, A., & Kurniawan, E. (2020). Rhetorical moves and linguistic complexity of research article abstracts in international applied linguistics journals. The Asian E.S.P. Journal, 16(5.2), 219-247.
Fitria, T. N. (2022). Structure Analysis of English Abstracts in International Journal Published in 2020. Journal of English Language and Culture, 12(1). https://doi.org/10.30813/jelc.v12i1.2815
Geng, H. (2024). Lexical Bundles in Rhetorical Moves of Introduction in English Linguistics Research Articles from Non-Scopus and Scopus Journals (Doctoral thesis, Universiti Putra Malaysia, Serdang, Malaysia).
Geng, H., & Nimehchisalem, V. (2023). Can ChatGPT Analyse Textual Data? The Sub-Themes Reflected by Typical Conceptual Metaphors in Short Stories of Language Assessment. ASEAN Journal of Applied Linguistics, 2, 16-31.
Geng, H., & Wei, H. (2023). Metadiscourse Markers in Abstracts of Linguistics and Literature Research Articles from Scopus-Indexed Journals. Journal of Modern Languages, 33(1), 29-49. https://doi.org/10.22452/jml.vol33no1.2
Geng, H., Lee, G. I., Jalaluddin, I., & Tan, H. (2023). Occurrence Frequency of Rhetorical Moves in Introductions of Linguistics Research Articles From Non-Scopus and Scopus Journals. Journal of Language Teaching and Research, 14(5), 1279-1289. https://doi.org/10.17507/jltr.1405.16
Geng, H., Lee, G. I., Jalaluddin, I., & Tan, H. (2023). Rhetorical Moves of Introduction Sections in English Linguistics Research Articles From Two Non-Scopus and Two Scopus Journals. Theory and Practice in Language Studies, 13(8), 2087-2096. https://doi.org/10.17507/tpls.1308.25
Ghasempour, B., & Farnia, M. (2017). Contrastive move analysis: Persian and English research article abstracts in law. The Journal of Teaching English for Specific and Academic Purposes, 5(4), 739-753. https://doi.org/10.22190/JTESAP1704739G
Gusmana, A. L. (2023). An Analysis Study of Rhetorical Moves in Sinta-Indexed Psychological Journal Abstracts [Doctoral dissertation] Universitas Pendidikan Indonesia.
Hyland, K. (2000). Disciplinary discourses: Social interaction in academic writing. Longman Pearson Education.
Joshi, A. (2016). Comparison between Scopus and ISI web of science. Journal Global Values, 7(1), 1-11.
Kanoksilapatham, B. (2005). Rhetorical structure of biochemistry research articles. English for specific purposes, 24(3), 269 292. https://doi.org/10.1016/j.esp.2004.08.003
Knight, S., Abel, S., Shibani, A., Goh, Y. K., Conijn, R., Gibson, A., ... & Shum, S. B. (2020). Are you being rhetorical? a description of rhetorical move annotation tools and open corpus of sample machine-annotated rhetorical moves. Journal of Learning Analytics, 7(3), 138-154. https://doi.org/10.18608/jla.2020.73.10
Kurniawan, E., & Sabila, N. A. A. (2021). Another look at the rhetorical moves and linguistic realizations in international and Indonesian journal articles: A case of tourism research article abstracts. Indonesian Journal of Applied Linguistics, 11(2), 318-329. https://doi.org/10.17509/ijal.v11i2.32055
Livberber, T. (2023). Toward non-human-centered design: Designing an academic article with ChatGPT. Profesional de la información/Information Professional, 32(5). https://doi.org/10.3145/epi.2023.sep.12
Lu, X., Casal, J. E., Liu, Y., Kisselev, O., & Yoon, J. (2021). The relationship between syntactic complexity and rhetorical move-steps in research article introductions: Variation among four social science and engineering disciplines. Journal of English for Academic Purposes, 54, 1-13. https://doi.org/10.1016/j.jeap.2021.101006
Rasmeenin, C. (2006). A structural move analysis of MA thesis discussion sections in applied linguistics. Mahidol University.
Reiss, M. V. (2023). Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. Preprints. https://doi.org/10.48550/arXiv.2304.11085
Saidi, M., & Karami, N. (2024). A Cross-Move Analysis of Interactional Metadiscourse Markers in Abstracts of Local and International Journals of History. Journal of Language Horizons, 7(4). https://doi.org/10.22051/lghor.2023.41767.1738
Santos, M. B. (1996). The textual organization of research paper abstracts in applied linguistics. Text- Interdisciplinary Journal for the Study of Discourse, 16(4), 481-499. https://doi.org/10.1515/text.1.1996.16.4.481
Savelka, J., Ashley, K. D., Gray, M. A., Westermann, H., & Xu, H. (2023). Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise? Preprints. https://doi.org/10.1145/3587102.3588792
Sidek, H. M. (2017). Entrepreneurial education conference proceedings: a rhetorical moves analysis of abstracts. IJASOS-International E-journal of Advances in Social Sciences, 3(9), 1112-1119. https://doi.org/10.18769/ijasos.401178
Siiman, L. A., Rannastu-Avalos, M., Pöysä-Tarhonen, J., Häkkinen, P., & Pedaste, M. (2023, August). Opportunities and Challenges for AI-Assisted Qualitative Data Analysis: An Example from Collaborative Problem-Solving Discourse Data. In International Conference on Innovative Technologies and Learning (pp. 87-96). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-40113-8_9
Swales, J. (1990). Genre analysis: English in academic and research settings. CUP.
Swales, J. M., & Feak, C. B. (2004). Academic writing for graduate students: Essential tasks and skills (Vol. 1). University of Michigan Press.
Tankó, G. (2017). Literary research article abstracts: An analysis of rhetorical moves and their linguistic realizations. Journal of English for Academic Purposes, 27, 42-55. https://doi.org/10.1016/j.jeap.2017.04.003
Zhang, B., & Wannaruk, A. (2016). Rhetorical Structure of Education Research Article Methods Sections. PASAA: Journal of Language Teaching and Learning in Thailand, 51, 155-184. https://doi.org/10.58837/CHULA.PASAA.51.1.6
Alyousef, H. S. (2023). A multidimensional analysis of linguistic realizations and rhetorical move structure in geography research article abstracts: A corpus-based study. Atlantis: Revista de la Asociación Española de Estudios Anglo-Norteamericanos, 45(1), 190-212. https://doi.org/10.28914/Atlantis-2023-45.1.11
Bhatia, V. K. (1993). Analyzing Genre: Language Use in Professional Settings. Longman Publishing.
Biber, D. (2007). Discourse on the move: Using corpus analysis to describe discourse structure (Vol. 28). John Benjamins Publishing. https://doi.org/10.1075/scl.28
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. Preprints. https://doi.org/10.48550/arXiv.2303.12712
Can, S., Karabacak, E., & Qin, J. (2016). Structure of moves in research article abstracts in applied linguistics. Publications, 4(3), 1-16. https://doi.org/10.3390/publications4030023
Ding, H. (2007). Genre analysis of personal statements: Analysis of moves in application essays to medical and dental schools. English for specific purposes, 26(3), 368-392. https://doi.org/10.1016/j.esp.2006.09.004
Fauzan, U., Lubis, A., & Kurniawan, E. (2020). Rhetorical moves and linguistic complexity of research article abstracts in international applied linguistics journals. The Asian E.S.P. Journal, 16(5.2), 219-247.
Fitria, T. N. (2022). Structure Analysis of English Abstracts in International Journal Published in 2020. Journal of English Language and Culture, 12(1). https://doi.org/10.30813/jelc.v12i1.2815
Geng, H. (2024). Lexical Bundles in Rhetorical Moves of Introduction in English Linguistics Research Articles from Non-Scopus and Scopus Journals (Doctoral thesis, Universiti Putra Malaysia, Serdang, Malaysia).
Geng, H., & Nimehchisalem, V. (2023). Can ChatGPT Analyse Textual Data? The Sub-Themes Reflected by Typical Conceptual Metaphors in Short Stories of Language Assessment. ASEAN Journal of Applied Linguistics, 2, 16-31.
Geng, H., & Wei, H. (2023). Metadiscourse Markers in Abstracts of Linguistics and Literature Research Articles from Scopus-Indexed Journals. Journal of Modern Languages, 33(1), 29-49. https://doi.org/10.22452/jml.vol33no1.2
Geng, H., Lee, G. I., Jalaluddin, I., & Tan, H. (2023). Occurrence Frequency of Rhetorical Moves in Introductions of Linguistics Research Articles From Non-Scopus and Scopus Journals. Journal of Language Teaching and Research, 14(5), 1279-1289. https://doi.org/10.17507/jltr.1405.16
Geng, H., Lee, G. I., Jalaluddin, I., & Tan, H. (2023). Rhetorical Moves of Introduction Sections in English Linguistics Research Articles From Two Non-Scopus and Two Scopus Journals. Theory and Practice in Language Studies, 13(8), 2087-2096. https://doi.org/10.17507/tpls.1308.25
Ghasempour, B., & Farnia, M. (2017). Contrastive move analysis: Persian and English research article abstracts in law. The Journal of Teaching English for Specific and Academic Purposes, 5(4), 739-753. https://doi.org/10.22190/JTESAP1704739G
Gusmana, A. L. (2023). An Analysis Study of Rhetorical Moves in Sinta-Indexed Psychological Journal Abstracts [Doctoral dissertation] Universitas Pendidikan Indonesia.
Hyland, K. (2000). Disciplinary discourses: Social interaction in academic writing. Longman Pearson Education.
Joshi, A. (2016). Comparison between Scopus and ISI web of science. Journal Global Values, 7(1), 1-11.
Kanoksilapatham, B. (2005). Rhetorical structure of biochemistry research articles. English for specific purposes, 24(3), 269 292. https://doi.org/10.1016/j.esp.2004.08.003
Knight, S., Abel, S., Shibani, A., Goh, Y. K., Conijn, R., Gibson, A., ... & Shum, S. B. (2020). Are you being rhetorical? a description of rhetorical move annotation tools and open corpus of sample machine-annotated rhetorical moves. Journal of Learning Analytics, 7(3), 138-154. https://doi.org/10.18608/jla.2020.73.10
Kurniawan, E., & Sabila, N. A. A. (2021). Another look at the rhetorical moves and linguistic realizations in international and Indonesian journal articles: A case of tourism research article abstracts. Indonesian Journal of Applied Linguistics, 11(2), 318-329. https://doi.org/10.17509/ijal.v11i2.32055
Livberber, T. (2023). Toward non-human-centered design: Designing an academic article with ChatGPT. Profesional de la información/Information Professional, 32(5). https://doi.org/10.3145/epi.2023.sep.12
Lu, X., Casal, J. E., Liu, Y., Kisselev, O., & Yoon, J. (2021). The relationship between syntactic complexity and rhetorical move-steps in research article introductions: Variation among four social science and engineering disciplines. Journal of English for Academic Purposes, 54, 1-13. https://doi.org/10.1016/j.jeap.2021.101006
Rasmeenin, C. (2006). A structural move analysis of MA thesis discussion sections in applied linguistics. Mahidol University.
Reiss, M. V. (2023). Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. Preprints. https://doi.org/10.48550/arXiv.2304.11085
Saidi, M., & Karami, N. (2024). A Cross-Move Analysis of Interactional Metadiscourse Markers in Abstracts of Local and International Journals of History. Journal of Language Horizons, 7(4). https://doi.org/10.22051/lghor.2023.41767.1738
Santos, M. B. (1996). The textual organization of research paper abstracts in applied linguistics. Text- Interdisciplinary Journal for the Study of Discourse, 16(4), 481-499. https://doi.org/10.1515/text.1.1996.16.4.481
Savelka, J., Ashley, K. D., Gray, M. A., Westermann, H., & Xu, H. (2023). Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise? Preprints. https://doi.org/10.1145/3587102.3588792
Sidek, H. M. (2017). Entrepreneurial education conference proceedings: a rhetorical moves analysis of abstracts. IJASOS-International E-journal of Advances in Social Sciences, 3(9), 1112-1119. https://doi.org/10.18769/ijasos.401178
Siiman, L. A., Rannastu-Avalos, M., Pöysä-Tarhonen, J., Häkkinen, P., & Pedaste, M. (2023, August). Opportunities and Challenges for AI-Assisted Qualitative Data Analysis: An Example from Collaborative Problem-Solving Discourse Data. In International Conference on Innovative Technologies and Learning (pp. 87-96). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-40113-8_9
Swales, J. (1990). Genre analysis: English in academic and research settings. CUP.
Swales, J. M., & Feak, C. B. (2004). Academic writing for graduate students: Essential tasks and skills (Vol. 1). University of Michigan Press.
Tankó, G. (2017). Literary research article abstracts: An analysis of rhetorical moves and their linguistic realizations. Journal of English for Academic Purposes, 27, 42-55. https://doi.org/10.1016/j.jeap.2017.04.003
Zhang, B., & Wannaruk, A. (2016). Rhetorical Structure of Education Research Article Methods Sections. PASAA: Journal of Language Teaching and Learning in Thailand, 51, 155-184. https://doi.org/10.58837/CHULA.PASAA.51.1.6
Published
2024-06-05
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).
