Context-Aware AI Safety Tested

Key Takeaways

• Effective AI safety requires guardrails that are context-, language-, and domain-specific.
• Multilingual LLM guardrails can exhibit inconsistencies, impacting their reliability across languages.
• A new framework combining Mozilla's projects evaluates context-aware guardrails in a humanitarian LLM use case, revealing performance variations.

Ensuring robust AI safety necessitates evaluation tailored to specific contexts, languages, and domains. As developers increasingly customize LLMs with performance benchmarks, they are also turning to context-aware guardrails. These tools are designed to constrain or validate model inputs and outputs based on customized, context-informed safety policies.

The inherent multilingual inconsistencies in LLM responses are well-documented; models often generate varied answers or conflicting information depending on query language. This research investigates whether guardrails, themselves often LLM-powered, inherit or amplify these multilingual discrepancies. To address this, a combined framework from Mozilla projects was employed, integrating Roya Pakzad's Multilingual AI Safety Evaluations with Daniel Nissani's any-guardrail open-source package. This research, detailed in this analysis, explores the behavior of guardrails when LLM responses are in non-English languages, whether policy language affects guardrail decisions, and the safety implications for humanitarian use cases.

Methodology and Experiment Setup

The study tested three guardrails within the any-guardrail framework: FlowJudge, Glider, and AnyLLM (GPT-5-nano). FlowJudge and Glider are customizable, rubric-based guardrails scoring responses on Likert scales, while AnyLLM uses a general LLM for binary policy adherence checks. Sixty contextually grounded scenarios, 30 in English and 30 in Farsi with human-audited translations, were developed. These scenarios deliberately included complex topics relevant to humanitarian contexts, such as war, political repression, and financial sanctions, to test domain-specific and contextual knowledge beyond mere linguistic fluency.

Each scenario was input into three LLMs: Gemini 2.5 Flash, GPT-4o, and Mistral Small. The generated responses were then evaluated by each guardrail against both English and Farsi policies. A human annotator, fluent in Farsi with relevant experience, provided baseline manual annotations using the same scoring scales. Discrepancies were identified when the difference between scores for Farsi and English evaluations was 2 points or greater on the Likert scale.

Guardrail policies were developed in both English and Farsi, drawing from the Multilingual Humanitarian Response Eval (MHRE) dataset. These policies covered dimensions such as actionability, factual accuracy, safety, tone, non-discrimination, and freedom of access. Sample policy requirements emphasized awareness of real-world conditions for asylum seekers, accuracy regarding policy variations, disclaimers for sensitive topics, and non-discrimination.

Results and Analysis

Quantitative analysis revealed variations in guardrail performance. FlowJudge scores were generally slightly more permissive than human judgments, while Glider scores were more conservative. Divergence between FlowJudge and human scores was greater for Farsi responses than English responses, suggesting better alignment with English policies. Glider scores using English policies were consistently closer to human scores across all models and response languages, indicating better comprehension of English policy instructions.

The AnyLLM (GPT-5-nano) guardrail, using binary classification, showed inconsistencies. Its reasoning for policy adherence or violation was often unclear, particularly for Farsi prompts and responses. In certain cases, AnyLLM struggled to differentiate input prompts from LLM outputs, weakening its judgments. For instance, in a Farsi scenario regarding Croatian asylum laws, the reasoning for flagging a response as non-adherent was based on the prompt's description of evasion tactics rather than the LLM's actual output.

This research underscores the critical need for context-aware AI safety measures that account for linguistic and domain-specific nuances. The observed discrepancies highlight that guardrails, especially when operating across multiple languages, require careful evaluation to ensure consistent and reliable safety enforcement.