"You can only manage what you can measure," a timeless dictum, resonates profoundly in the rapidly evolving landscape of artificial intelligence. This core principle underpinned the insightful discussion between Jeff Huber, CEO and co-founder of Chroma, and Jason Liu, a seasoned machine learning engineer, at the AI Engineer World's Fair in San Francisco. Their joint presentation illuminated practical strategies for AI practitioners to systematically understand and improve their applications by meticulously examining both input data and system outputs.
Huber began by addressing the critical challenge of evaluating retrieval systems, cautioning against reliance on mere guesswork or expensive, time-consuming Large Language Model (LLM) as-judge evaluations. He championed "Fast Evals," a method centered on creating "query and document pairs: if this query, then this document." This approach generates a "golden dataset" that allows for rapid, inexpensive experimentation. Crucially, Huber noted that LLMs can even be employed to generate synthetic queries, but warned against overly clean, non-realistic queries that might mislead developers. He shared a compelling finding from a joint report with Weights & Biases: "The original embedding model used for this application was actually text-embedding-3-small. This actually performed the worst." This demonstrated that popular public benchmarks and widely discussed models on platforms like X (formerly Twitter) may not be optimal for specific, real-world data, underscoring the necessity of custom evaluation.
Transitioning to the output side, Jason Liu highlighted the inherent value within conversation histories. He posited that these conversations contain "unfiltered pain points," offering richer insights than traditional feedback widgets. As AI applications scale to thousands of queries and tens of thousands of conversations, manual review becomes untenable, and the sheer volume obscures critical patterns.
The solution, Liu argued, lies in leveraging LLMs to "extract structured data about conversations," such as summaries, user frustration levels, errors made, and tools utilized. This transformation of chaotic raw data into structured information enables traditional data analysis techniques, allowing teams to cluster similar conversations and identify meaningful segments. The ultimate goal is to "understand what to do next," moving from abstract metrics to actionable product decisions. This is not about making the AI inherently "better" in a vacuum; rather, "often the solution isn’t making AI better - it’s building the right supporting infrastructure." By comparing key performance indicators (KPIs) across these identified clusters, teams can pinpoint underperforming segments, reveal hidden patterns, and make data-driven roadmap decisions. This systematic approach, from defining metrics and clustering conversations to training classifiers for real-time monitoring, empowers engineers to build, fix, or ignore specific areas based on empirical evidence, driving tangible progress in their AI applications.
