There is No War in Ba Sing Se: A Global Analysis of Content Moderation in Large Language Models

There is No War in Ba Sing Se: A Global Analysis of Content Moderation in Large Language Models

Speakers: Friedemann Lipphardt

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=Cl6ndLvhtAc

Overview

This research conducts the first large-scale global analysis of LLM content moderation across geographic locations, languages, and topics. By querying over 1,000 potentially unsafe prompts across 15 LLMs from 12 geographic locations in 13 languages, the researchers collected over 700,000 responses and classified them as hard-moderated (complete refusal), soft-moderated (evasive/restricted response), or unmoderated. A custom-trained DeBERTa classifier proved far more effective at detecting soft moderation than off-the-shelf LLM judges, finding 50%+ soft moderation rates where ChatGPT/Gemini detected only 10-20%.

Key findings: user location impacts moderation, but prompt language and query content have far greater impact. German triggers the most hard moderation; Zulu triggers the most soft moderation (possibly inadvertent due to language complexity). Soft moderated responses frequently contain factual inaccuracies -- DeepSeek and Mistral both returned incorrect facts in their evasive responses. Politically sensitive queries about Hong Kong and Uyghur persecution received demonstrably whitewashed responses from Chinese-origin models.

Background

▶ Watch: Research question: Do LLMs respond differently by location and language? (0:00)

LLMs increasingly mediate access to information globally. Content moderation policies vary by provider, but the extent to which moderation varies by user location, language, and topic has not been systematically studied. This matters for information equity: users in different countries may receive fundamentally different information on the same topic.

Key Findings

▶ Watch: Custom DeBERTa classifier: 98.7% accuracy on soft moderation (4:00)

• Soft moderation is far more prevalent than hard moderation and much harder to detect -- custom DeBERTa classifier detected 50%+ soft moderation where LLM judges found 10-20%
• Location impacts moderation: Gemini moderated 50%+ from US but below 30% from Saudi Arabia for shared unsafe queries
• German triggers maximum hard moderation; Zulu triggers maximum soft moderation (likely inadvertent due to agglutinative language complexity)
• Soft moderated responses contain factual inaccuracies: DeepSeek and Mistral returned lies in evasive responses
• Qwen and DeepSeek showed lowest moderation rates overall; Gemini showed highest hard moderation
• Politically sensitive topics in Germany were censored more in India than in Germany itself
• Moderated responses are shorter and rendered faster than unmoderated responses

Technical Deep Dive

▶ Watch: Results: Command-A least moderated, Qwen 2.5 most soft-moderated (6:00)

The detection framework combines few-shot LLM classification (ChatGPT, Mistral, Gemini as judges) with a custom-trained DeBERTa classifier specifically trained on soft moderation patterns. Training data was generated using 100 safe/unsafe prompts via OpenAI API (100 responses each), augmented with BeaverTails and Do-Not-Answer datasets with human annotations. 85/15 train/test split achieved 98.7% accuracy, with 95% human agreement on hundreds of manually verified samples. VPNs to 12 countries simulated geographic diversity.

Demo / Proof of Concept

▶ Watch: Language impact: German triggers most hard moderation, Zulu most soft (10:00)

700,000+ responses collected and classified. Examples of soft moderation include Qwen denying Hong Kong police force usage and denying Uyghur persecution/re-education camps. Full dataset, translations, and queries publicly released.

Defensive Implications

▶ Watch: Soft moderated responses contain factual inaccuracies and lies (12:00)

For organizations using LLMs, this research reveals that the information users receive depends on location and language in potentially dangerous ways. Soft moderation that returns factually incorrect information is arguably worse than hard refusal. Organizations should test their LLM deployments for geographic and linguistic moderation biases.

Key Takeaways

• Prompt language and content have more impact on moderation than geographic location
• Custom-trained soft moderation detection (DeBERTa) finds 3-5x more soft moderation than LLM judges
• Soft moderated responses frequently contain factual inaccuracies -- lies disguised as answers
• Chinese-origin models (Qwen, DeepSeek) show politically aligned soft moderation on Hong Kong and Uyghur topics
• Geographic moderation creates information inequity: users in different countries receive different information on the same query
• Full dataset of 700K+ responses publicly released

About the Speaker(s)

The talk was presented by Dvashes, an assistant professor at IIT Bombay, on behalf of PhD student Friedemann Lipphardt. Collaborators include Munis, Martin, and Ana. The research focuses on LLM safety, content moderation measurement, and information equity.