Safeguarding Anonymized Data in the Age of Language Models: Challenges and Solutions

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Sep 27 2023

In a paper published in the journal Scientific Reports, researchers evaluated anonymization methods in the context of the potential of artificial intelligence (AI) to re-identify individuals from remaining information. They conducted experiments using a Generative Pre-trained Transformer (GPT) on anonymized texts of famous figures. Developing a novel methodology employing Large Language Models (LLMs) enhanced text anonymity.

Background

Privacy concerns have led to text anonymization methods like Textwash in today's data-driven society. However, challenges remain in balancing privacy and data utility. Protecting individuals' identities in anonymized texts while preserving meaningful content is paramount. Striking this delicate balance is a fundamental challenge in text anonymization.

Past studies have addressed personal data, identifiers, and text anonymization algorithms. Notably, the General Data Protection Regulation (GDPR) defines personal data and emphasizes the importance of data protection techniques like anonymization. Various text anonymization methods have been explored based on named entity recognition and machine learning (ML). Textwash utilizes ML and contextualized word representations to redact sensitive information while maintaining semantic coherence. GPT-3 plays a crucial role in natural language processing (NLP), where GPT-3 raises ethical concerns, prompting discussions on fairness, accountability, and transparency in AI. The development of moral and transparent AI models is essential, as reflected in recent regulatory initiatives such as the European Union (EU) to regulate AI technologies EU AI)Act.

Risk Scenario

GitHub's Copilot, powered by OpenAI's Codex, recently encountered security issues. These incidents underscore the potential of LLM to comprehend sensitive information. Training LLMs on extensive sensitive datasets could pose deanonymization threats, as they may infer missing information from anonymized texts. In a hypothetical scenario, two organizations exchange anonymized data, but one organization's use of LLMs poses a risk of deanonymizing the other's data. This issue emphasizes the need for safeguarding anonymized data against LLM-based attacks.

Experimental Findings on Deanonymization and Anonymization

The authors utilized a dataset containing anonymized descriptions of famous individuals in their experiments. This dataset was initially collected by instructing participants to write reports of celebrities, which were then anonymized using Textwash. To assess the effectiveness of LLMs in deanonymization and anonymization tasks, the researchers performed various experiments with this dataset.

In the first set of experiments, the investigation focused on the deanonymization capabilities of GPT-3.5. The results showed that GPT successfully deanonymized a significant portion of the anonymized texts and even outperformed humans in identifying the described celebrities. They also discovered some misclassifications, which revealed the potential for further improvements. Additionally, they explored a method called "Hide in Plain Sight" by assessing its effectiveness in mitigating deanonymization risks.

The second set of experiments focused on anonymization by comparing the anonymization performance of GPT with Textwash. The results showed that GPT was more efficient at identifying sensitive tokens and performed slightly better than Textwash regarding deanonymization. They also highlighted the specific tokens exclusively captured by GPT, demonstrating its capacity to identify salient information that could lead to deanonymization. These experiments collectively raised concerns about the limitations of current text anonymization techniques in the face of advanced LLMs. They emphasized the need for improved anonymization methodologies to protect individual privacy effectively.

Privacy Risks in Language Models

Researchers have increasingly explored the privacy implications of language models in recent years. Numerous studies have examined Bidirectional Encoder Representations from Transformers (BERT) capacity to reveal sensitive information. They concluded that it could establish associations without a significant threat. Notably, even including patient names in training data did not significantly elevate these risks.

Other approaches acknowledged the privacy risks posed by LLMs by stressing the importance of training them with data intended for public use rather than relying on publicly available data. The approach aligns with Zimmer's perspective. The perspective emphasizes that data published by individuals may only account for some potential uses and consequences. They also pointed out that the differential privacy method is more practical for such language models. GPT-2, another technique, extracted personally identifiable information (PII) from anonymized texts. However, their focus was on generic PII, and their threat scenario differed from the one addressed in this work. They also did not offer a comparison with human capabilities.

Conclusion

To sum up, this work sheds light on the ethical concerns surrounding the use of AI in re-identifying individuals from remaining information. Companies leverage legally collected data to personalize services, potentially using LLMs like GPT for document deanonymization. Experiments demonstrate that LLMs pose a significant threat, surpassing human capabilities in deanonymization. The need for new privacy protection strategies has led to considering approaches such as introducing misleading cues and strategically replacing PII to deceive LLM responses.

Developing LLM-oriented metrics and focusing on sensitive text parts will be crucial in the evolving field of text anonymization. Additionally, utilizing context-aware methods and generative AI will enhance privacy protection strategies. In the coming years, LLMs will play a central role in attacking and defending against such threats.