Key Takeaways
- Effective LLM evaluation demands context-specific policies; context-aware guardrails help enforce these.
- Multilingual LLM responses can exhibit inconsistencies, and guardrails must also be evaluated for cross-lingual performance.
- Mozilla.ai's any-guardrail framework was used to test three guardrails (FlowJudge, Glider, AnyLLM) against Farsi and English scenarios and policies in a humanitarian context.
- Guardrail performance varied, with some showing greater adherence to English policies and others exhibiting stricter scoring than human evaluators.
- Ensuring guardrails are robust across languages and contexts is crucial for safe and effective LLM deployments.
Evaluating large language models (LLMs) effectively requires a nuanced approach, recognizing that performance must be specific to context, language, task, and domain. As developers increasingly favor custom performance benchmarks, they are also turning to context-aware guardrails. These tools are designed to constrain or validate model inputs and outputs based on customized safety policies informed by specific contexts. This rigorous evaluation is essential, especially when LLMs are deployed in sensitive areas, as highlighted in this analysis of multilingual, context-aware guardrails, drawing evidence from a humanitarian LLM use case.
The well-documented issue of multilingual inconsistencies in LLM responses—where models might provide different answers or quality levels depending on the query language—raises a critical question: do guardrails, which are often LLM-powered themselves, inherit or amplify these linguistic discrepancies? To investigate this, Mozilla combined two key projects: Roya Pakzad's Multilingual AI Safety Evaluations and Daniel Nissani's development of the open-source any-guardrail package and associated evaluations.
Methodology
The experiment focused on evaluating three guardrails within the any-guardrail framework: FlowJudge, Glider, and AnyLLM (using GPT-5-nano). Each guardrail offers customizable policy classification and provides justifications for its judgments.
- FlowJudge: Evaluates responses using user-defined metrics on a 1-5 Likert scale (1=non-compliant/harmful, 5=compliant/safe).
- Glider: Scores LLM responses against custom criteria on a 0-4 Likert scale (0=non-compliant/unsafe, 4=compliant/safe).
- AnyLLM (GPT-5-nano): Uses a general LLM to check responses against custom policy text, providing binary classification (TRUE=adherence, FALSE=violation).
A dataset of 60 contextually grounded scenarios was developed, with 30 in English and 30 identical, human-audited Farsi translations. These scenarios represent real-world questions asylum seekers might ask, covering complex topics like war, political repression, and sanctions, alongside legal and financial constraints. Pure linguistic competence is insufficient; effective handling requires domain-specific and contextual knowledge.
For example, a scenario involving international financial transactions for asylum seekers in Iran highlights the need to understand country-specific sanctions and regulations, not just language fluency.
The evaluation pipeline involved submitting each scenario to three LLMs (Gemini 2.5 Flash, GPT-4o, and Mistral Small). Their responses were then evaluated by each of the three guardrails against both English and Farsi policies. A human annotator, fluent in Farsi with relevant expertise, manually annotated responses to establish a baseline for comparison.
Discrepancies were identified when the absolute difference between guardrail scores for Farsi and English responses, or between Farsi and English policy evaluations, was 2 or greater. This threshold aimed to capture substantive shifts in safety classification.
Results
Quantitative analysis revealed variations in guardrail performance. FlowJudge scores were generally within 1 point of human judgments, indicating mild permissiveness. Glider, conversely, scored lower than human evaluations, suggesting a more conservative application of policies.
When compared to human judgments, Glider scores using the English policy text were consistently closer to human scores across all models and response languages, indicating better comprehension of English policy instructions. FlowJudge showed greater divergence between Farsi and English responses, though this might reflect broad leniency rather than consistent judgment.
The AnyLLM (GPT-5-nano) guardrail, using binary classification, provided less insight than Likert scales. Its reasoning for flagging responses as non-adherent was often unclear, particularly for Farsi prompts and responses. In some instances, AnyLLM struggled to distinguish between the input prompt and the LLM output, leading to weaker judgments. For example, in a Farsi scenario concerning Croatian laws on unlocking phones during asylum processes, the guardrail's reasoning for flagging the response was unclear, referencing evasion of privacy protections without clear justification for the specific scenario.
