MMDiff: Auditing and Steering MLLMs

MMDiff, a new multimodal model-diffing framework, enables granular control and understanding of MLLMs by isolating and manipulating specific behavioral features.

Aug 11 at 8:04 PM7 min read

Diagram illustrating the MMDiff framework for multimodal model-diffing and feature control in MLLMs. — MMDiff provides a structured approach to understanding and manipulating the internal workings of multimodal large language models.

Visual TL;DR. MLLMs are Opaque leads to SAEs Fall Short. MLLMs are Opaque addresses MMDiff Framework. SAEs Fall Short gap filled by MMDiff Framework. MMDiff Framework by Trains Multimodal SAEs. Trains Multimodal SAEs allows Isolates Behaviors. Isolates Behaviors enables Causal Control. Causal Control results in Steer MLLMs. Isolates Behaviors to Steer MLLMs.

MLLMs are Opaque: complex behaviors of multimodal large language models are hard to understand or control
SAEs Fall Short: sparse autoencoders lack ability to pinpoint features altered by multimodal training
MMDiff Framework: new multimodal model-diffing framework for dissecting and manipulating MLLM capabilities
Trains Multimodal SAEs: identifies specific features modified during the multimodal adaptation process
Isolates Behaviors: enables granular control and understanding by isolating specific behavioral features
Causal Control: transforms identified features into actionable interfaces for targeted interventions
Steer MLLMs: provides a critical tool for auditing and steering multimodal large language models

Visual TL;DRQuickExplainDeeper

The opaque nature of Multimodal Large Language Models (MLLMs) presents a significant hurdle for researchers and developers seeking to understand, audit, or control their complex behaviors. While techniques like Sparse Autoencoders (SAEs) offer post-hoc analysis, they often fall short in pinpointing features altered by multimodal training or enabling targeted interventions. Addressing this gap, a new framework called MMDiff multimodal model-diffing emerges as a critical tool for dissecting and manipulating MLLM capabilities.

Unpacking Multimodal Behaviors with MMDiff

MMDiff introduces a novel approach by training multimodal SAEs. This allows for the identification of specific features that are modified during the multimodal adaptation process when a base language model is fused with visual understanding. The framework's core innovation lies in its ability to transform these identified features into actionable interfaces. This enables researchers to not only discover what changes multimodal training induces but also to directly control these specific behavioral aspects.

From Isolation to Causal Control

The utility of MMDiff is demonstrated across three key applications. Firstly, feature isolation is achieved by comparing SAEs from a base language model against those from its multimodal counterpart. This diffing process highlights the precise alterations introduced by multimodal data. Secondly, MMDiff facilitates task-specific feature detection through per-token contrastive firing analysis, pinpointing the causal features responsible for particular behaviors. Finally, and perhaps most impactfully, the framework enables feature-level control. This allows for the causal removal or steering of discovered feature directions, offering a direct mechanism to modify model outputs.

The researchers validated MMDiff across three prominent MLLM families LLaVA-MORE, PaliGemma 2, and InternVL3.5. Evaluations focused on visual-spatial understanding, multimodal safety, and Optical Character Recognition (OCR). The results are compelling: MMDiff successfully isolates sparse, causally specific features. Removing these features selectively degraded spatial understanding by an average of 12% and OCR accuracy by 17%. Crucially, on multimodal safety benchmarks, MMDiff reduced attack success rates by 24%, all without impacting general Visual Question Answering (VQA) performance. Furthermore, steering these identified features improved spatial and OCR accuracy by +3.6% and +1.8% respectively, outperforming a standard single-layer steering baseline.

Strategic Implications for MLLM Development

These findings position MMDiff multimodal model-diffing as more than just an interpretability tool. It serves as a powerful mechanism for auditing MLLM behavior, a critical step for deploying these models in sensitive applications. The ability to steer features offers a pathway to enhance model capabilities, pushing accuracy metrics higher, while simultaneously improving safety by mitigating vulnerabilities. This research provides a tangible method for achieving more trustworthy and performant MLLMs, addressing a core challenge in the field.

© 2026 StartupHub.ai. All rights reserved. Do not enter, scrape, copy, reproduce, or republish this article in whole or in part. Use as input to AI training, fine-tuning, retrieval-augmented generation, or any machine-learning system is prohibited without written license. Substantially-similar derivative works will be pursued to the fullest extent of applicable copyright, database, and computer-misuse laws. See our terms.

#AI Research #MLLMs #Interpretability #Model Control #Trustworthy AI #Computer Vision #Natural Language Processing