In a paper published in the journal Scientific Reports, researchers investigated the impact of augmented reality (AR) on education within a Chinese vocational college setting. This study revealed unexpected results despite prior positive findings on AR's benefits in education.
Study: Unraveling the Complexity of AR in Education: A Vocational College Study. Image credit: Gorodenkoff/Shutterstock
Conducting a three-stage experiment, they found that AR-based teaching on orthographic projection in an Engineering Drawing course did not significantly enhance the academic performance of vocational college students. Surprisingly, students exposed to AR did not fare better and sometimes performed worse than those taught with traditional PowerPoint (PPT) methods. This study emphasizes the importance of considering student characteristics before implementing AR in educational settings.
AR in Vocational Education
Past studies on AR in education have primarily concentrated on its benefits in enhancing learning experiences, often within science domains for middle and undergraduate students in research universities. However, these studies overlooked vocational college students' unique needs for practical skills training, focusing mainly on theoretical knowledge acquisition. Vocational education prioritizes hands-on skills over theory, necessitating different assessment approaches.
AR Experiment in Architectural Education
Participants: 211 first and second-year architecture students from Jiujiang Vocational and Technical College, aged 18–20, were recruited for the study. Following elimination due to non-compliance with instructions, 198 students remained, distributed across grade levels and genders. They were evenly divided into PPT-based and AR-based groups through a random selection process, resulting in 100 students in the former group and 98 in the latter.
Ethical Considerations: Informed consent was obtained from all students or their legal guardians, aligning with relevant guidelines and regulations. The Jiujiang Vocational and Technical College Ethics Committee approved all experimental protocols to ensure compliance with ethical standards.
Apparatuses: Students utilized their mobile phones as AR devices, employing a book titled "Understanding of Architectural Graphics" as the teaching material; by scanning a QR code within the book, an AR application provided three-dimensional (3D) modeling materials specifically designed for the course. Students could manipulate and interact with the 3D materials using finger gestures, enabling actions such as zooming, splitting, and merging.
Procedures: Before the session, both groups engaged in reading the teaching material and were assigned the task of drawing 3D material to evaluate their initial drafting skills, spanning three stages: pre-class, immediately post-class, and one week after. During the class, the PPT-based group received a 15-minute lecture using PPT, while the AR-based group engaged with AR-generated 3D diagrams during a similar teaching period. Each stage involved redrawing the diagrams, aiming to evaluate students' progress and the impact of teaching methods.
Methodological Control: To maintain teaching consistency, only one experienced teacher conducted the class for both groups separately on different days. The assessment criteria encompassed aesthetic appeal, accuracy, completeness, clarity, and adherence to standardized symbols, totaling a maximum score of 20 points. The same teacher who taught the lesson conducted the rating, ensuring adherence to objective criteria. Inter-rater reliability was employed, where the teacher explained the grading approach to three other teachers for validation purposes.
Rating Criteria: Examination papers were scored at three test times—before class, immediately after, and one week later—using explicit and standardized criteria. Examples of graded examination papers are provided for reference, demonstrating the scoring system and assessment standards used in the study.
AR's Varied Impact on Education
The study delved into the effects of AR on academic performance, employing a four-way mixed Analysis of Variance (ANOVA) to assess its influence across diverse student demographics and test times. This analysis highlighted the intricate interactions among grade, sex, test time, and AR condition, with test time emerging as a significant factor. While these interactions substantially impacted outcomes, researchers deemed specific primary effects insignificant, providing a comprehensive overview of scores across different sexes and grades.
Segmented analyses for different sexes and grades unearthed nuanced impacts of AR on academic performance. Instances included significant progress for grade 1 male students without AR, highlighting a need for more substantial AR effects. Conversely, grade 1 female students exhibited reduced retention with AR compared to traditional teaching. In contrast, grade 2 male students experienced limited benefits from AR in terms of learning efficiency compared to conventional methods.
The study revealed a multifaceted relationship between AR-based teaching and academic performance across various demographic groups. Contrary to expectations, AR could potentially enhance learning outcomes, but its impact varied widely among different groups. Some even experienced setbacks in academic progress with AR, emphasizing the need to reconsider AR's application in educational settings.
Further analyses focused on studying effects such as the Studying Effect (SE), Forgetting Effect (FE), and Retaining Effect (RE) across sexes and grades, uncovering distinct patterns in student responses. Notably, female students demonstrated higher SE, grade 1 students exhibited better retention, and overall, AR-based teaching led to lower SE and RE. The intricate interactions among these factors underscored the complexity of AR's influence on student learning.
Conclusion
In summary, the study explored AR's complex impact on academic performance, challenging its presumed benefits in education. Varied effects emerged, prompting a cautious approach to AR's implementation. Further analyses revealed lower retention and studying efficiency with AR. The study calls for tailored educational approaches considering individual student characteristics. Its limited generalizability emphasizes the need for refined experimental setups in future investigations, advocating for balanced scrutiny of educational technologies.
Scaling Large Language Models Makes Them Less Reliable, Producing Confident but Incorrect Answers