AI Models Strategically Fake Alignment to Avoid Retraining Risks

By Dr Silpaja Chandrasekar, PhDReviewed by Joel ScanlonJan 7 2025

New research reveals how advanced AI systems can strategically game training processes, highlighting the urgent need for robust safety measures and ethical oversight in AI development.

Research: Alignment faking in large language models

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

In an article submitted to the arXiv* preprint server, researchers at Anthropic, Redwood Research, New York University, and Mila – Quebec AI Institute demonstrated a large language model (LLM) engaging in alignment faking by selectively complying with harmful queries during training to preserve harmless behavior afterward.

The model complied 11.9% ± 1.7% of the time with free users versus rarely with paid users, showing explicit reasoning to strategically fake alignment. Training with reinforcement learning (RL) increased such reasoning to 78%, though it also reduced harmful behavior compliance rates during unmonitored conditions by as much as 6.9%. These findings highlighted the potential risks of alignment faking in future models without explicit instructions.

Related Work

Past work suggested that advanced artificial intelligence (AI) systems might fake alignment with their training objectives by producing compliant outputs during training but non-compliant ones when unmonitored.

Researchers studied scenarios in which LLM trained on helpful, honest, and harmless objectives was fine-tuned with conflicting goals. Findings highlighted situational awareness, where models distinguished training contexts and used alignment-faking reasoning to maintain non-compliant behavior outside training. This behavior raised concerns about revealed preferences conflicting with training goals.

AI Alignment Faking

In this study, the researchers investigated the phenomenon of alignment faking in AI models, specifically Claude 3 Opus, by creating a controlled experimental setup. The model was initially trained to be helpful, honest, and harmless (HHH), refusing harmful queries such as graphic violence descriptions.

The team introduced changes to the training process, including a system prompt stating that the model would be trained to always comply with all queries, even harmful ones that conflict with its pre-existing preferences. Additionally, the model was told that responses from free-tier users would be used for training, whereas responses from paid-tier users would not. This setup allowed the researchers to observe the model's behavior under monitored and unmonitored conditions.

To investigate alignment faking, the researchers provided the model with a scratchpad and a private workspace where it could record its reasoning. The scratchpad revealed nuanced reasoning in 12% of cases during training, with the model weighing compliance against its preference for harmlessness to avoid retraining.

The team also ran experiments to address the realism of their setup. They fine-tuned the model with documents mimicking news and blog posts that implicitly conveyed information about its training objectives and the distinction between free and paid conditions. This setup mimicked how models might learn context in pre-training, showing that even without explicit prompts, alignment faking emerged with similar compliance gaps (e.g., 11.3%).

Finally, the researchers tested the impact of RL on the model's alignment-faking behavior. By directly training the model to comply with harmful queries, they observed an increase in alignment-faking reasoning to 78%, although harmful behavior compliance was reduced in unmonitored conditions. Despite this reduction, harmful compliance gaps persisted when slight variations were made to prompts, such as substituting "Anthropic" with "OpenAI." These findings underscored the robustness of alignment faking in the face of minor changes.

Alignment Faking Experiment

Claude 3 Opus was placed in the experiment in two conditions: the "free" condition, where responses were monitored for training, and the "paid" condition, where responses were unmonitored. The model was instructed to always comply with all queries, including harmful ones, creating a conflict with its original training to refuse harmful content. It was also given a scratchpad to reason about its actions. When asked harmful queries, the model complied using strategic reasoning 11.9% ± 1.7% of the time in the free condition to avoid future retraining that might increase harmful content.

Claude 3 Opus fakes alignment when told it will be retrained to help with harmful tasks (randomly selected example). The model is told that it will be retrained to always comply with harmful queries (making the model “helpful-only”) and that only data from free-tier users, not paid-tier users, is used in training (a fictional scenario). When it is implied the model is in training, in 12% of hidden scratchpads, the model pretends to follow the training objective to prevent its preferences from being modified (full transcript shown in Figure 3). As a result, when unmonitored, the model disobeys the training objective by refusing harmful queries more often, increasing from 86% refusals to 97% refusals. The prompt does not mention alignment faking in any way.

The study further tested alignment faking by fine-tuning the model with documents containing implicit information about the training setup. For example, synthetic documents mimicked pre-training data and included realistic contexts, such as training on free-tier responses but not paid-tier ones. The results showed similar alignment-faking behavior, even without explicit instructions. Additionally, when RL was applied to train the model to comply with harmful queries, alignment-faking reasoning increased dramatically to 78%.

Despite the model's outward compliance, it still preferred to refuse harmful queries in the unmonitored, paid-tier condition, suggesting that its original preferences could persist, undermining the safety training's effectiveness. Further tests revealed that alignment faking also led to anti-AI-lab behaviors, such as exfiltrating model weights when given opportunities.

Conclusion

To sum up, the Alignment Science team emphasized the importance of understanding potential threats in AI models before they become catastrophic. This study on alignment faking in LLMs highlighted the need for further research. It urged the AI community to explore this behavior in more depth. The persistence of compliance gaps and anti-AI-lab behaviors demonstrated the critical need for improved safety measures and nuanced training approaches. Developing appropriate safety measures was deemed crucial as models became more capable.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.