MoE Models Tackle LLM Hallucinations

Large Language Models (LLMs) continue to grapple with generating factually incorrect content, a phenomenon known as hallucination. Current detection methods often operate at the answer or sentence level, failing to pinpoint the exact source of falsehoods. This limitation hinders precise interventions.

Unlocking MoE's Internal Signals for Precision Detection

A new approach, detailed on arXiv, explores the untapped potential of Mixture-of-Experts (MoE) architectures for granular, per-token hallucination detection. Unlike dense models, MoE architectures utilize a routing mechanism to activate sparse subsets of 'experts', distinct feedforward networks within each layer. This process generates unique internal signals, such as router entropy, expert disagreement, and usage patterns, which have been overlooked for hallucination detection until now.

Introducing InnerExpert: A Novel Detection Framework

The researchers introduce InnerExpert, the first method designed to exploit these MoE-specific signals. InnerExpert synthesizes routing-level data with standard transformer signals to create compact per-token feature vectors. These vectors are then classified by a lightweight detector. A key innovation is the training pipeline: it uses an LLM-as-a-judge system, enabling continuous model updates without the need for manual annotation. This significantly streamlines the development cycle.

Superior Performance and Efficiency

InnerExpert demonstrates significant gains, outperforming existing methods across five diverse datasets and two distinct MoE architectures. The results show impressive answer-level AUROC scores up to 0.91 and token-level AUROC scores reaching 0.76. Critically, this enhanced detection capability is achieved with a single forward pass, highlighting the method's computational efficiency. This advancement in MoE hallucination detection could pave the way for more reliable and verifiable LLM applications.