Visual TL;DR
Video MLLM InefficiencyDriver
processing adjacent frames as independent images leads to redundant tokens
From the articleThe inherent temporal redundancy in video, where adjacent frames largely overlap, presents a fundamental inefficiency for current video multimodal large language models (video MLLMs).
Temporal RedundancyContext
adjacent video frames largely overlap, causing inflated computational costs
From the articleThe inherent temporal redundancy in video, where adjacent frames largely overlap, presents a fundamental inefficiency for current video multimodal large language models (video MLLMs).
AdaCodec IntroducedCore
a new dynamic and efficient video interface for MLLMs
From the article 3 mentionsInstead of encoding every frame fully, this system, instantiated as AdaCodec, selectively transmits a full reference frame only when scene prediction is unreliable.
Predictive Visual CodingCore
intelligently manages visual token transmission based on scene prediction
From the articleThe core innovation lies in a 'predictive visual code' that intelligently manages visual token transmission.
Selective Frame EncodingContext
From the articleInstead of encoding every frame fully, this system, instantiated as AdaCodec, selectively transmits a full reference frame only when scene prediction is unreliable.
Compact P-tokensCore
From the articleOtherwise, it encodes inter-frame changes, encompassing motion and prediction residuals, using compact 'P-tokens'.
Reduced Token CountEffect
significantly minimizes visual tokens required for video understanding
From the articleEven at a drastically reduced token budget (1/7th), AdaCodec with 32k tokens outperforms the 224k baseline on all long-video benchmarks.
Efficiency GainsOutcome
drastically cuts tokenization costs and latency for video MLLMs
From the articleThis efficiency leap makes real-time video analysis and interaction far more feasible.
Superior PerformanceOutcome
achieves better results at a fraction of the computational budget