Test-Time Distillation Nearly Doubles Model Performance

Traditional model distillation focuses on transferring capabilities by updating a smaller model's parameters during training. However, a new approach explores whether this transfer can occur effectively at inference time, a concept termed strong-to-weak scaffolding. This method allows a stronger 'builder' model to create inference-time 'harnesses' that guide a weaker 'target' model to solve tasks more reliably, entirely bypassing parameter updates.

Scaffolding Reasoning at Inference Time

Researchers demonstrated this test-time capability transfer using four Theory-of-Mind benchmarks. A builder model iteratively refined its harness over several rounds, using a small validation set. The finalized harness then guided the target model on the full test set. The results were striking: average target-model performance nearly doubled, jumping from 0.49 to 0.91. This significant improvement highlights the power of guiding inference processes externally.

Deconstructing Performance Gains

Analysis revealed that the substantial performance boost is not due to encouraging the target model to reason more extensively or sample more broadly. Instead, the gains primarily originate from offloading unstable model reasoning into deterministic code, implementing benchmark-specific routing logic, and strictly enforcing answer formats. This suggests that the builder model's contribution is more about structuring and constraining the inference process than about enhancing the target model's intrinsic reasoning capacity.

Strategic Implications for Capability Transfer

The study found that builder-model reasoning effort correlates monotonically with harness quality. While platform effects are modest compared to the builder model's inherent capability, weaker target models benefit the most from this inference-time capability transfer. This work introduces inference-time harness design as a potent complement to conventional training-time distillation. It offers a novel pathway for strong models to imbue weaker counterparts with cognitive structure and improve task performance without the need for retraining, as detailed in the original paper.