Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study

    Research output: Contribution to journalArticlepeer-review

    2 Scopus citations

    Abstract

    With the immense interest in ChatGPT worldwide, education has seen a mix of both excitement and skepticism. To properly evaluate its impact on education, it is crucial to understand how far it can help students without prior knowledge answer assessment questions. This study aims to address this question as well as the impact of the question type. We conducted multiple experiments with computer engineering students (experiment group: n = 41 to 56), who were asked to use ChatGPT to answer previous test questions before learning about the related topics. Their scores were then compared with the scores of previous-term students who answered the same questions in a quiz or exam setting (control group: n = 24 to 61). The results showed a wide range of effect sizes, from −2.55 to 1.23, depending on the question type and content. The experiment group performed best answering code analysis and conceptual questions but struggled with code completion and questions that involved images. However, the performance in code generation tasks was inconsistent. Overall, the ChatGPT group’s answers lagged slightly behind the control group’s answers with an effect size of −0.16. We conclude that ChatGPT, at least in the field of this study, is not yet ready to rely on by students who do not have sufficient background to evaluate generated answers. We suggest that educators try using ChatGPT and educate students on effective questioning techniques and how to assess the generated responses. This study provides insights into the capabilities and limitations of ChatGPT in education and informs future research and development. © 2023 Copyright held by the owner/author(s).
    Original languageAmerican English
    JournalACM Transactions on Computing Education
    Volume4
    Issue number45
    StatePublished - 2023

    Keywords

    • ChatGPT
    • large language models
    • Education computing
    • Control groups
    • Effect size
    • Empirical studies
    • Knowledge use
    • Language model
    • Large language model
    • Prior-knowledge
    • Question type
    • Worldwide education
    • Students

    Fingerprint

    Dive into the research topics of 'Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study'. Together they form a unique fingerprint.

    Cite this