Sina Shahandeh, from Radicait, shared insights into the challenges and methodologies for running autonomous agents in scientific tasks. He highlighted that while current AI agents are adept at implementation and running experiments, they often hit a plateau when faced with the creative demands of scientific discovery, specifically in generating novel hypotheses.
The Hypothesis Generation Bottleneck
Shahandeh explained that in scientific endeavors, humans excel at observing, questioning, hypothesizing, and iterating. While AI agents can efficiently handle the memory and implementation aspects, the critical step of generating a good hypothesis remains a significant bottleneck. He illustrated this with an example from Radicait's work on generating in-silico PET scans from CT scans, a task that involves complex image translation. Initial attempts by agents to optimize the model led to saturation, indicating a need for more than just brute-force experimentation.
Decomposing Problems for AI Agents
To overcome this, Shahandeh proposed a method of breaking down complex, long-horizon scientific problems into smaller, manageable components. This hierarchical decomposition, visualized as a component map, allows agents to focus on specific aspects of the problem. He demonstrated how this structure, when generated through a prompt, can guide an LLM to systematically explore and suggest improvements for each component. This approach is analogous to how human researchers approach complex problems.
The Power of Hierarchical Documentation
Shahandeh showcased how such a hierarchical structure can be documented, with each component linked to its code. This creates a navigable knowledge base that agents can utilize to generate hypotheses. For instance, in the CT to PET scan generation example, the agent could be prompted to suggest changes to the model architecture, data preprocessing, or loss functions based on this structured understanding.
Adversarial and Collaborative Loops
The presentation emphasized the potential for using multiple agents in an adversarial or collaborative loop to refine hypotheses. This process mirrors human scientific collaboration, where different perspectives and critiques can lead to more robust solutions. He also touched upon integrating specialized skills, such as multimodal image understanding from models like Gemini, into the agent's workflow to enhance its observational and analytical capabilities.