Optimizing VLM Pretraining Data Mixes

The current landscape of Vision Language Model (VLM) pretraining is plagued by heuristic data curation. Practitioners often stack datasets based on ad-hoc quality filters and intuitive domain ratios, lacking a principled approach for new data inclusion. This opacity hinders reproducible advancements and leaves frontier recipes undisclosed. To address this, researchers have introduced DecoupleMix, a framework that transforms data construction into a systematic mixture-optimization problem, establishing it as a reproducible engineering discipline.

Decomposing the Data Mixture Challenge

DecoupleMix achieves this by decoupling the complex mixture problem into two orthogonal sub-problems: inter-class ratios across different capabilities and intra-class ratios within a specific category. Inter-class allocation is managed through a single-variable iterative search, while intra-class composition employs a multidimensional, dataset-level assessment. This assessment scores datasets on Quality and Difficulty, formulating selection as a constrained convex optimization problem with a diversity objective.

Enabling Attributable and Guided Data Curation

This systematic approach yields two critical capabilities. Firstly, it provides clear guidance on what data to collect next, moving beyond guesswork. Secondly, it renders dataset validation a controlled, attributable experiment. Experiments demonstrate that DecoupleMix consistently surpasses heuristic baselines. Notably, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without requiring retuning. Furthermore, by utilizing 80B additional multimodal continue-pretraining tokens, a VLM trained with this method proved competitive against strong open-source models trained with substantially larger multimodal budgets, highlighting the efficiency gains achievable in Vision Language Model pretraining.