Cutting-Edge AI & VR System Helps Students with Autism Navigate Social Life

Researchers at the University of Kansas are revolutionizing social skills training with AI-enhanced VR. Their new system, iKNOW, uses extended reality and real-time AI interactions to provide students with disabilities a safe, engaging way to practice and master essential social behaviors.

Research: The Social Validity of Video Modeling Versus Virtual Reality for Improving the Social Communication Skills of Middle School Students. Image Credit: Kansas University

For more than a decade, University of Kansas researchers have been developing a virtual reality system to help students with disabilities, especially those with autism spectrum disorder, learn, practice, and improve the social skills they need during a typical school day. The KU research team has now secured funding to add artificial intelligence components to the system, giving those students an extended reality, or XR, experience to sharpen social interactions in a more natural setting.

The U.S. Office of Special Education Programs has awarded a five-year, $2.5 million grant to researchers at KU's School of Education & Human Sciences to develop Increasing Knowledge and Natural Opportunities With Social Emotional Competence, or iKNOW. The system will build on previous work and provide students and teachers with an immersive, authentic experience that blends extended reality with real-world elements of artificial intelligence.

iKNOW will expand the capabilities of VOISS, a Virtual reality Opportunity to Integrate Social Skills. This KU-developed VR system has proven successful and statistically valid in helping students with disabilities improve their social skills. That system contains 140 unique learning scenarios meant to teach knowledge and understanding of 183 social skills in virtual school environments such as a classroom, hallway, cafeteria, or bus that students and teachers can use via multiple platforms such as iPad, Chromebooks, or Oculus VR headsets. The system also helps students use social skills such as receptive or expressive communication across multiple environments, not simply in the isolation of a classroom.

IKNOW will combine the VR aspects of VOISS with AI features such as large language models to enhance the systems' capabilities and allow more natural interactions than listening to prerecorded narratives and responding by pushing buttons. The new system will enable user-initiated speaking responses that can accurately transcribe spoken language in real time. The AI technology of iKNOW will also be able to generate appropriate video responses to avatars students interact with, audio analysis of user responses, and integration of in-time images and graphics with instruction to boost students' contextual understanding.

"Avatars in iKNOW can have certain reactions and behaviors based on what we want them to do. They can model the practices we want students to see," said Amber Rowland, assistant research professor in the Center for Research on Learning, part of KU's Life Span Institute and one of the grant's co-principal investigators. "The system will harness AI to make sure students have more natural interactions and put them in the role of the 'human in the loop' by allowing them to speak, and it will respond like a normal conversation."

The spoken responses will be more natural and relatable to everyday situations, and the contextual understanding cues will help students better understand why a certain response is preferred. Rowland said that when students were presented with multiple choices in previous versions, they often knew which answer was correct, but they indicated that's not how they would have responded in real life.

IKNOW will also provide a real-time student progress monitoring system, which will tell educators and families how long and frequently students speak, the number of keywords used, where students may have struggled in the system, and other data to help enhance understanding.

All avatar voices that iKNOW users encounter are provided by real middle school students, educators, and administrators. This helps enhance the system's natural environment without the shortcomings of students practicing social skills with classmates in supervised sessions. For example, users do not have to worry about what the people they are practicing with think about them while they are learning. They can practice the social skills they need until they are comfortable moving from the XR environment to real life.

"It will leverage our ability to take something off of teachers' plates and provide tools for students to learn these skills in multiple environments. Right now, the closest we can come to that is training peers. But that puts students with disabilities in a different box by saying, 'You don't know how to do this,'" said Maggie Mosher, assistant research professor in KU's Achievement & Assessment Institute, a co-principal investigator for the grant.

Mosher, a KU graduate who completed her doctoral dissertation comparing VOISS to other social skills interventions, found that the system was statistically significant and valid in improving social skills and knowledge across multiple domains. Her study, which also found the system acceptable, appropriate, and feasible, was published in the high-impact journals Computers & Education and Issues and Trends in Learning Technologies.

The grant supporting iKNOW is one of four OSEP Innovation and Development grants intended to spur innovation in educational technology. The research team, including principal investigator Sean Smith, professor of special education; Amber Rowland, associate research professor in the Center for Research on Learning and the Achievement & Assessment Institute; Maggie Mosher, assistant research professor in AAI; and Bruce Frey, professor in educational psychology, will present their work on the project at the annual I/ITSEC conference, the world's largest modeling, simulation, and training event. It is sponsored by the National Training & Simulation Association, which promotes international and interdisciplinary cooperation within the fields of modeling and simulation, training, education, and analysis. It is also affiliated with the National Defense Industrial Association.

The research team has implemented VOISS on the Apple Store and Google Play at schools nationwide. The iKNOW site offers information, demonstrations, and videos, and users can contact developers to use the system at the "work with us" page.

IKNOW will add resources for teachers and families who want to implement the system to a website called iKNOW TOOLS (Teaching Occasions and Opportunities for Learning Supports) to help them generalize social skills across real-world settings.

"By combining our research-based social emotional virtual reality work (VOISS) with the increasing power and flexibility of AI, iKNOW will further personalize the learning experience for individuals with disabilities along with the struggling classmates," Smith said. "Our hope and expectation is that iKNOW will further engage students to develop the essential social emotional skills to then apply in the real world to improve their overall learning outcomes."