Heterogeneous LLMs Unite for Performance

The pursuit of enhanced LLM capabilities often centers on scaling monolithic models. However, a novel approach detailed on arXiv demonstrates that composing heterogeneous, frozen large language models can unlock significant performance gains with surprising efficiency.

Unified Latent Space for Heterogeneous Models

This research introduces a feedforward graph architecture where diverse, pre-trained LLMs function as computational nodes. These models communicate by projecting their representations into a shared, continuous latent space. This is achieved through learned linear projections, building on prior work that showed geometric compatibility between independently trained LLM latent spaces. The innovation lies in extending this compatibility to an end-to-end trainable multi-node graph structure. Projection matrices are optimized jointly via backpropagation, utilizing residual stream injection hooks to enable gradient flow across multiple frozen model boundaries.

Efficient Composition Yields Superior Performance

The architecture effectively integrates smaller frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) for input encoding into a shared latent space. This aggregate signal is then fed into larger frozen models (Phi-3-mini, Mistral-7B) whose representations are processed by a lightweight cross-attention output node. Remarkably, this sophisticated multi-model LLM architecture boasts only 17.6 million trainable parameters, dwarfed by the approximately 12 billion parameters of the frozen constituent models. Despite this parameter disparity, the system achieves state-of-the-art results on benchmarks like ARC-Challenge (87.3%), OpenBookQA (82.8%), and MMLU (67.2%). These scores surpass the best single constituent model by substantial margins (up to 11.4 percentage points) and outperform parameter-matched learned classifiers on frozen single models.

Emergent Routing and Tractable Gradients

Empirical validation confirms that gradient flow through multiple frozen model boundaries is tractable. Furthermore, the output node exhibits selective routing behavior across the second-layer nodes without any explicit supervisory signals, suggesting emergent intelligence in task decomposition and information routing within this novel multi-model LLM architecture.