In a paper published in the journal Proceedings of the National Academy of Sciences, researchers tackled the challenge of socially situated learning for AI agents by formalizing this as a reinforcement learning problem, where agents learned to ask meaningful questions via social interactions. The research involved an interactive agent improving visual intelligence by asking natural language questions about photos on a social network.
Study: AI Agents Learn and Adapt through Social Interaction. Image credit: sdecoret/Shutterstock
Unlike traditional methods, this agent adapted its behavior based on observed social norms, leading to a 112% improvement in recognizing new visual information. In a controlled field experiment, it outperformed an active-learning baseline by 25.6%, offering promising developments for AI agents in open social environments.
Background
Researchers liken current methods for training artificial intelligence (AI) agents to isolating each agent in a room with books. These agents rely on vast amounts of manually labeled training data or web content for learning, leading to significant progress in various fields. However, when a concept is absent in their training data, these agents need more means to acquire it. This limitation restricts their knowledge of what's available in the room, making them ill-equipped to handle novel situations or adapt to evolving real-world scenarios. Consequently, although these agents often perform well on test sets, help is needed when faced with new challenges or real-world deployment.
Related Work
Past studies in this domain have focused on enabling AI agents to transcend the confines of their metaphorical training rooms. The concept of "socially situated artificial intelligence" empowers AI agents to broaden their knowledge by actively interacting with individuals in real-world social environments.
Drawing inspiration from human development, where children acquire knowledge and cultural norms through dialogues with more knowledgeable members of society, this approach has promising applications in domains that necessitate effective human-AI interactions, including human-computer interaction, interactive robotics, personalized conversational agents, and accessible technology. Previous approaches to learning from human interaction predominantly depended on human labels generated manually or operated within domains with restricted action spaces, like games and simulations featuring a limited set of moves.
Proposed Method
Optimizing the sequence of labeling requests to expand a model's capabilities efficiently characterizes the active learning framework. The ultimate objective is to enhance the model's performance using newly acquired data while minimizing the number of requests made. While many active-learning methods employ heuristic acquisition algorithms, recent endeavors have formalized this process as a reinforcement-learning paradigm. However, these attempts typically remove real humans from the loop and assume the existence of an oracle capable of providing labels for any request.
Nevertheless, a pure active-learning approach aimed at gathering data through social interactions faces challenges, as recent research in human-computer interaction reveals that users are reluctant to act as simple oracles by repeatedly supplying labels. This limitation points to a significant issue: Traditional active learning is not ecologically valid within realistic social environments. Field experiments demonstrate that a baseline active-learning approach generates interactions that people could be more interested in responding to. A novel concept termed "socially situated AI" is introduced to overcome these challenges.
This approach formulates an iterative reinforcement-learning problem, extending conventional active learning. It places the agent within a social environment, denoted as the state of the domain (e.g., history of dialogues for conversational agents or current object locations for robotic agents). The interaction space, depicting possible interactions with people initiated by the agent, plays a pivotal role. Transition dynamics encode how people react to the agent's past actions and how the world evolves. Additionally, the probability measure of the initial state distribution is a crucial element of this framework.
In this socially situated learning paradigm, the agent seeks to balance two primary objectives: interacting to learn and learning to interact effectively with people. A carefully designed reward function encompasses interaction rewards, encouraging interactions aligned with community norms, and knowledge rewards that promote interactions yielding data that significantly enhance the model's performance. The agent's decision-making process conforms to an infinite-horizon Markov decision process, where meta-transitions govern the evolution of environment states, data history, and the model's capabilities. These interactions contribute to the dataset, leading to model training and updates. The agent's performance evaluation considers model accuracy and the rate of informative interactions, reflecting its understanding of social norms and the effectiveness of its interactions. This framework provides a flexible foundation applicable to various domains, including conversational assistants, educational agents, and assistive robotics.
Experimental Analysis
In evaluating the socially situated AI framework, two key goals for the agent involve initiating social interactions that people find engaging and responsive. Simultaneously, the other objective centers on enhancing the underlying model by obtaining valuable data, employing these goals as evaluation metrics. The informative response rate, indicating the percentage of interactions that receive answers, measures the agent's ability to garner responses. The agent can recognize new visual concepts by reporting accuracy on a test set of social media images, questions, and answers. The comparative analysis involves a baseline approach and two ablations of the socially situated agent, which isolate the effects of interaction and knowledge rewards. The socially situated agent significantly improves informative response rate and model accuracy during an 8-month deployment, outperforming other methods and revealing its potential in social environments.
Conclusion
To sum up, the framework enables AI agents to learn collaboratively with people in social environments, breaking free from isolated training scenarios. The field study demonstrated its feasibility, offering opportunities for applications like healthcare support robots and technology interfaces that learn from diverse communities to enhance their capabilities and understanding.
Reinforcement Learning Boosts Factory Layout Optimization