# MMDiff: Auditing and Steering MLLMs
_MMDiff, a new multimodal model-diffing framework, enables granular control and understanding of MLLMs by isolating and manipulating specific behavioral features._
**Published:** 2026-08-11
**Source:** https://www.startuphub.ai/ai-news/ai-research/2026/mmdiff-auditing-and-steering-mllms
---
The opaque nature of Multimodal Large Language Models (MLLMs) presents a significant hurdle for researchers and developers seeking to understand, audit, or control their complex behaviors. While techniques like Sparse Autoencoders (SAEs) offer post-hoc analysis, they often fall short in pinpointing features altered by multimodal training or enabling targeted interventions. Addressing this gap, a new framework called [MMDiff multimodal model-diffing](https://arxiv.org/abs/2608.09928v1) emerges as a critical tool for dissecting and manipulating MLLM capabilities.

MLLMs are OpaqueDriver

From the article 5 mentionsThe opaque nature of Multimodal Large Language Models (MLLMs) presents a significant hurdle for researchers and developers seeking to understand, audit, or control their complex behaviors.leads toSAEs Fall ShortDriversparse autoencoders lack ability to pinpoint features altered by multimodal trainingFrom the articleWhile techniques like Sparse Autoencoders (SAEs) offer post-hoc analysis, they often fall short in pinpointing features altered by multimodal training or enabling targeted interventions.gap filled byMMDiff FrameworkCoreFrom the article 9+ mentionsAddressing this gap, a new framework called MMDiff multimodal model-diffing emerges as a critical tool for dissecting and manipulating MLLM capabilities.byTrains Multimodal SAEsContextidentifies specific features modified during the multimodal adaptation processFrom the article 3 mentionsMMDiff introduces a novel approach by training multimodal SAEs.allowsIsolates BehaviorsEffectenables granular control and understanding by isolating specific behavioral featuresFrom the article 4 mentionsThe results are compelling: MMDiff successfully isolates sparse, causally specific features.enablesCausal ControlEffecttransforms identified features into actionable interfaces for targeted interventionsFrom the article 5 mentionsSecondly, MMDiff facilitates task-specific feature detection through per-token contrastive firing analysis, pinpointing the causal features responsible for particular behaviors.results inSteer MLLMsOutcomeprovides a critical tool for auditing and steering multimodal large language modelsFrom the article 6 mentionsThe ability to steer features offers a pathway to enhance model capabilities, pushing accuracy metrics higher, while simultaneously improving safety by mitigating vulnerabilities.

## Unpacking Multimodal Behaviors with MMDiff

MMDiff introduces a novel approach by training multimodal SAEs. This allows for the identification of specific features that are modified during the multimodal adaptation process when a base language model is fused with visual understanding. The framework's core innovation lies in its ability to transform these identified features into actionable interfaces. This enables researchers to not only discover what changes multimodal training induces but also to directly control these specific behavioral aspects.

## From Isolation to Causal Control

The utility of MMDiff is demonstrated across three key applications. Firstly, feature isolation is achieved by comparing SAEs from a base language model against those from its multimodal counterpart. This diffing process highlights the precise alterations introduced by multimodal data. Secondly, MMDiff facilitates task-specific feature detection through per-token contrastive firing analysis, pinpointing the causal features responsible for particular behaviors. Finally, and perhaps most impactfully, the framework enables feature-level control. This allows for the causal removal or steering of discovered feature directions, offering a direct mechanism to modify model outputs.

The researchers validated MMDiff across three prominent MLLM families LLaVA-MORE, PaliGemma 2, and InternVL3.5. Evaluations focused on visual-spatial understanding, multimodal safety, and Optical Character Recognition (OCR). The results are compelling: MMDiff successfully isolates sparse, causally specific features. Removing these features selectively degraded spatial understanding by an average of 12% and OCR accuracy by 17%. Crucially, on multimodal safety benchmarks, MMDiff reduced attack success rates by 24%, all without impacting general Visual Question Answering (VQA) performance. Furthermore, steering these identified features improved spatial and OCR accuracy by +3.6% and +1.8% respectively, outperforming a standard single-layer steering baseline.

## Strategic Implications for MLLM Development

These findings position MMDiff multimodal model-diffing as more than just an interpretability tool. It serves as a powerful mechanism for auditing MLLM behavior, a critical step for deploying these models in sensitive applications. The ability to steer features offers a pathway to enhance model capabilities, pushing accuracy metrics higher, while simultaneously improving safety by mitigating vulnerabilities. This research provides a tangible method for achieving more trustworthy and performant MLLMs, addressing a core challenge in the field.
---
Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.