MMDiff: Auditing and Steering MLLMs

The opaque nature of Multimodal Large Language Models (MLLMs) presents a significant hurdle for researchers and developers seeking to understand, audit, or control their complex behaviors. While techniques like Sparse Autoencoders (SAEs) offer post-hoc analysis, they often fall short in pinpointing features altered by multimodal training or enabling targeted interventions. Addressing this gap, a new framework called MMDiff multimodal model-diffing emerges as a critical tool for dissecting and manipulating MLLM capabilities.

Unpacking Multimodal Behaviors with MMDiff

MMDiff introduces a novel approach by training multimodal SAEs. This allows for the identification of specific features that are modified during the multimodal adaptation process when a base language model is fused with visual understanding. The framework's core innovation lies in its ability to transform these identified features into actionable interfaces. This enables researchers to not only discover what changes multimodal training induces but also to directly control these specific behavioral aspects.

From Isolation to Causal Control

The utility of MMDiff is demonstrated across three key applications. Firstly, feature isolation is achieved by comparing SAEs from a base language model against those from its multimodal counterpart. This diffing process highlights the precise alterations introduced by multimodal data. Secondly, MMDiff facilitates task-specific feature detection through per-token contrastive firing analysis, pinpointing the causal features responsible for particular behaviors. Finally, and perhaps most impactfully, the framework enables feature-level control. This allows for the causal removal or steering of discovered feature directions, offering a direct mechanism to modify model outputs.

The researchers validated MMDiff across three prominent MLLM families LLaVA-MORE, PaliGemma 2, and InternVL3.5. Evaluations focused on visual-spatial understanding, multimodal safety, and Optical Character Recognition (OCR). The results are compelling: MMDiff successfully isolates sparse, causally specific features. Removing these features selectively degraded spatial understanding by an average of 12% and OCR accuracy by 17%. Crucially, on multimodal safety benchmarks, MMDiff reduced attack success rates by 24%, all without impacting general Visual Question Answering (VQA) performance. Furthermore, steering these identified features improved spatial and OCR accuracy by +3.6% and +1.8% respectively, outperforming a standard single-layer steering baseline.

Strategic Implications for MLLM Development

These findings position MMDiff multimodal model-diffing as more than just an interpretability tool. It serves as a powerful mechanism for auditing MLLM behavior, a critical step for deploying these models in sensitive applications. The ability to steer features offers a pathway to enhance model capabilities, pushing accuracy metrics higher, while simultaneously improving safety by mitigating vulnerabilities. This research provides a tangible method for achieving more trustworthy and performant MLLMs, addressing a core challenge in the field.