ChatGPT Earns a B in Engineering Course but Fails at Critical Thinking

Aerospace engineering researchers tested ChatGPT as a "virtual student"; it passed with a B, but only by acing easy questions and flunking the hard ones. Can AI truly learn, or just look smart?

Melkior Ornik lecturing in AE 353: Aerospace Control Systems.  Research: The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

With the assumption that students are going to use artificial intelligence and large language models such as ChatGPT to do their homework, researchers in the Department of Aerospace Engineering in The Grainger College of Engineering, University of Illinois Urbana-Champaign set out to learn how well the free version of ChatGPT would compare with human students in a semester-long undergraduate control systems course.

The results: On straightforward math homework, ChatGPT got an A, but with some quirky answers. However, on higher-level problems that require reasoning, it got a D.

“We found ChatGPT technology can get an A on structured, straightforward questions. On open-ended questions it got a 62, brining ChatGPT’s semester grade down to an 82, a low B. The class average for the human students was 84.85 percent because they could handle the problems that required higher-level reasoning,” said Ph.D. student Gokul Puthumanaillam.

The study concludes that a student who puts in minimal effort, showing no interest in learning the material, could use ChatGPT exclusively and still earn a B and pass the course. The problem is that the passing grade might be a combination of an A+ in simple math and a D- in analysis. They haven’t learned much.

“Like calculators in math classes, ChatGPT is a tool that’s here to stay and that students will use. What the results of this study pointed out to me is that I need to adjust as an educator,” said Puthumanaillam’s advisor, Melkior Ornik. “I plan to consider how I design my courses so that, over time, I include more higher-level questions, perhaps including project-based assignments. Students will still use programs like ChatGPT to do the simpler math problems, but by adding more open-ended questions, they’ll also reach a higher level of critical thinking and truly learn the material.”

Puthumanaillam said that although ChatGPT is fast and mostly correct on structured questions, it’s wise to use it with caution.

“A student might take 20 minutes to answer a question. ChatGPT solves it in less than 20 seconds, but the correctness is sometimes questionable.”

He also described examples of unusual behavior from ChatGPT’s homework, such as using inappropriate technical jargon and stating things that were not true.

“Despite the fact that we provided all of the course material needed to ChatGPT, it still hallucinated, using words like quasi periodic oscillations that were never used in the class, in the lectures or course materials.”

The study considered the type of student who chooses to put in minimal effort. Puthumanaillam said the premium version of ChatGPT may be slightly more capable of solving the analytical questions and can store more memory to solve longer, more complex problems. The researchers chose to use the free version of the software because the average student probably wouldn’t want to spend the monthly fee.

Puthumanaillam noted that there were no team assignments in the course; the ChatGPT prompts used the exact language that human students received, and all interactions were conducted synchronously.

“When the students were doing their homework, ChatGPT was also doing the homework. ChatGPT was just an extra student in the class.”

Did ChatGPT learn from its mistakes?

“When we told ChatGPT it was wrong on a multiple-choice question, gave it the correct option, then a variation of the same question, yes, it did better. In a sense, it was learning but overall, it was stagnant. If it scored 90 percent in homework, it ended up scoring 90 or 92 at the conclusion of the semester.”

This work was supported by the Grants for Advancement of Teaching in Engineering program at The Grainger College of Engineering, University of Illinois, Urbana-Champaign. Aerospace Professor Tim Bretl, along with Ph.D. students Grayson Schaer and Pranay Thangeda, created the project environments, developed course material, and the PrairieLearn infrastructure.

The study, “The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own,” by Gokul Puthumanaillam and Melkior Ornik, is available online and will be presented at the 14th International Federation of Automatic Control Symposium on Advances in Control Education in June. DOI: 10.48550/arXiv.2503.05760

The syllabus and course materials for AE 353: Aerospace Control Systems, as well as sample prompts used in the research, are available online.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.