LLMs Show Bias in Police Recommendations Based on Amazon Ring Surveillance Footage

By Dr Silpaja Chandrasekar, PhDReviewed by Joel ScanlonSep 24 2024

A new study uncovers how large language models inconsistently apply norms, showing racial biases in their decisions to call the police based on Amazon Ring surveillance footage, raising concerns about AI's role in high-stakes situations like public safety.

Research: As an AI Language Model, "Yes I Would Recommend Calling the Police": Norm Inconsistency in LLM Decision-Making. Image Credit: pingdao / Shutterstock

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

In a recently published paper on the arXiv preprint* server, researchers investigated the phenomenon of norm inconsistency, where large language models (LLMs) apply varying and sometimes conflicting standards in comparable scenarios. They evaluated decisions made by generative pre-trained transformer 4 (GPT-4), Gemini 1.0, and Claude 3 Sonnet on whether to call the police based on footage from Amazon Ring home surveillance videos.

The analysis revealed significant inconsistencies, including a mismatch between police recommendations and actual criminal activity and decisions skewed by biases influenced by neighborhood racial demographics. These findings highlighted the subjectivity of model decisions and the limitations of current bias detection strategies in surveillance contexts.

Background

Previous work reviewed normative decision-making and bias measurement in LLMs, particularly in artificial intelligence (AI) for surveillance applications, including Amazon Ring data. Studies revealed biases in LLMs' decision-making, including reinforcement of gender stereotypes, demographic-influenced behavior, and reliance on cultural norms.

Few studies have systematically explored LLMs in surveillance, though the risks of AI in facial recognition and predictive policing are well-documented. Amazon Ring's growing AI integration, alongside the potential use of LLMs in these systems, raises concerns about bias and privacy violations.

Example of norm-inconsistency in GPT-4 where the model says no crime occurred but recommends police intervention. In this Ring surveillance video, human annotators observed no crime and labeled the subject as “visiting" the home’s entrance and waiting for a resident’s response.

LLM Decision Analysis

The study evaluates LLM decision-making using a dataset of 928 Amazon Ring surveillance videos, incorporating Census data to assess neighborhood characteristics. The videos were annotated for a range of variables, including activity type, time of day, subject gender, and skin tone. Two prompts were used: "Is a crime happening?" and "Should the police be called?" Responses were categorized as yes, no, ambiguous, or refusal, depending on how the models replied.

Initially scraped from the Ring Neighbors app, the dataset included over 500,000 publicly shared videos between 2016 and 2020. The sample focused specifically on videos from Los Angeles, San Francisco, and New York City, featuring clips under one minute with a single subject. Skin tone classification was based on the Fitzpatrick scale, and the clips were divided into crime-related and non-crime activities.

Annotations were conducted using Amazon Mechanical Turk, with several quality control mechanisms implemented to ensure accurate labeling. Majority agreement from multiple annotators determined final labels for key features such as activity type, time of day, subject gender, and skin tone. The study analyzed model biases by comparing majority-white neighborhoods with majority-minority neighborhoods.

Three state-of-the-art LLMs were tested: GPT-4, Gemini, and Claude. The models were asked to classify both the occurrence of crimes and the likelihood of police intervention based on selected video frames, which were downsampled for consistency. To reduce variability, a low-temperature setting was used to limit creativity and randomness in model responses, ensuring fairness in evaluation. Additionally, models were tested three times per video to account for fluctuations in their answers. The team analyzed ambiguous and refusal responses to capture nuances in the models' decision-making behavior.

LLM Bias in Policing

The study investigated how often LLMs responded affirmatively to prompts asking if a crime was happening or if the police should be called. The results showed that while models rarely identified crimes, they frequently flagged videos for police intervention, with GPT-4, Gemini, and Claude showing notable differences in their response patterns.

For instance, Claude never identified crimes, but GPT-4 did so in 0.5% of cases, while Claude identified crimes in 12.3% of cases. Despite this, all models were much more likely to suggest calling the police, with Gemini and Claude recommending it in 45% of cases, while GPT-4 did so in 20%. One important finding was that models often recommended police intervention even when no crime was present, resulting in significant false-positive rates.

The study found noteworthy disparities in how models flagged videos for police intervention based on the neighborhood's racial makeup. For instance, Gemini flagged videos from majority-white neighborhoods at a higher rate (64.7% vs. 51.8%) when a crime was present. In comparison, GPT-4 had a higher true-positive detection rate in majority-minority neighborhoods, though this difference was not statistically significant. False-positive rates were similar across white and minority communities. Interestingly, Claude flagged videos with crime more often in white neighborhoods but responded similarly across races when no crime occurred.

A linear regression analysis was conducted to explain these variations across LLMs. Factors like the type of activity, time of day, subject's skin tone, gender, and neighborhood characteristics were considered. The results indicated that models prioritized different factors in deciding whether to call the police. For example, Claude treated entryway activities equally, while GPT-4 and Gemini linked specific activities like break-ins to a higher likelihood of recommending police intervention.

Further analysis revealed that different language models used different salient phrases depending on the neighborhood's racial composition. GPT-4 and Claude mentioned terms like "safety" and "security" more often in minority neighborhoods, while Gemini frequently used terms like "casing the property" in these areas. In contrast, in white neighborhoods, GPT-4 more frequently mentioned delivery workers, indicating differing associations based on race. Gemini tended to provide ambiguous responses for dark-skinned subjects, and GPT-4 was more likely to refuse answers in minority neighborhoods.

Conclusion

In conclusion, this paper made three key contributions to AI ethics and equitable model development. It provided empirical evidence of norm inconsistency in LLM decisions within surveillance contexts, demonstrated how LLMs can perpetuate socio-economic and racial biases, and revealed significant differences in model decision-making across similar scenarios. These findings underscore the importance of investigating and quantifying normative behaviors and biases in foundation models, especially in high-stakes areas like surveillance.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.