# Langfuse: Domain Expertise Crucial for AI Self-Improvement _Langfuse's Annabelle Schäfer explains why domain expertise is crucial for AI self-improvement, advocating for high-signal target functions and expert-driven data._ **Published:** 2026-07-18 **Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/langfuse-domain-expertise-crucial-for-ai-self-improvement --- In the rapidly evolving AI landscape, the drive towards self-improving systems is paramount. However, achieving this requires more than just sophisticated algorithms; it demands a deep understanding of the specific domain in which the AI operates. Annabelle Schäfer, Growth Engineer at Langfuse, a leading open-source observability and evaluation platform for AI systems, recently highlighted this critical point in a presentation titled "Stop Burning Tokens: Why self-improvement needs domain expertise first." Schäfer emphasized that while the industry buzz is all about "loops" and "auto-optimization," the success of these systems hinges on meticulously defining their goals and ensuring high-signal feedback mechanisms. AI Self-ImprovementContext industry buzz about 'loops' and 'auto-optimization' for AI agentsFrom the articleSchäfer outlined a three-step approach to building effective self-improvement systems:hindered byAmbiguous TargetsDriverrelying solely on prompt engineering leads to poorly defined AI goalsFrom the article 7 mentionsA key challenge, Schäfer explained, lies in defining clear target functions, especially for AI applications in specialized domains like healthcare or legal compliance.requiresDomain ExpertiseCoreAnnabelle Schäfer advocates for deep understanding of specific AI operating domainsFrom the article 4 mentionsBy focusing on domain expertise and high-signal feedback, developers can build more reliable and performant AI applications.High-Signal FeedbackEffectmeticulously defining goals and ensuring clear, quantifiable feedback mechanismsFrom the article 4 mentionsSchäfer then addressed the challenge of translating this "right/wrong" high-signal feedback to other, less deterministic AI applications.Langfuse PlatformCoreopen-source observability and evaluation platform for AI systemsFrom the article 3 mentionsTheir platform supports a workflow that includes tracing, monitoring, building datasets, experimenting, and evaluating AI outputs.needsExpert-Driven DataEffectcrucial for translating high-signal feedback to diverse AI applicationsFrom the article 2 mentionsShe stressed that great target functions require sufficient data volume and high-signal evaluators, often developed through collaboration with domain experts.contributes toSuccessful AI LoopsOutcomeachieving robust, self-improving AI systems through clear objectivesFrom the article 4 mentionsSchäfer began by referencing prominent figures in the AI space, such as Boris Journy and Peter Steinberger, who advocate for designing "loops" rather than relying solely on prompt engineering for AI agents. ## The Challenge of Ambiguous Targets Schäfer began by referencing prominent figures in the AI space, such as Boris Journy and Peter Steinberger, who advocate for designing "loops" rather than relying solely on prompt engineering for AI agents. She noted that the recent surge in interest around auto-research and auto-improvement topics, while promising, often stems from a developer-centric perspective. A key challenge, Schäfer explained, lies in defining clear target functions, especially for AI applications in specialized domains like healthcare or legal compliance. Unlike coding, where a clear binary outcome like "does it compile" serves as a straightforward target, many AI applications deal with far more nuanced and subjective goals. She illustrated this with a diagram showing a winding path from a "start" point to an "optimal destination," often obscured by challenges like storms and rough terrain. This visual metaphor underscores that the initial targets given to AI agents are inherently incomplete, and discovering the true optimal path requires iterative refinement and a deep understanding of the problem space. ## Langfuse's Approach to Improvement Loops Langfuse, Schäfer explained, is building the infrastructure to facilitate these improvement loops, not just for humans but also for AI agents themselves. Their platform supports a workflow that includes tracing, monitoring, building datasets, experimenting, and evaluating AI outputs. The core idea is to move from "know what's going on" to "find what's interesting/wrong," then to "capture what you want to work," "test hypothesis," and finally "check if it worked." This iterative process, when done correctly, allows teams to continuously upgrade and improve their applications, shipping with confidence. ## The Power of Clear, Quantifiable Feedback Schäfer detailed an experiment conducted by Langfuse to identify the clearest possible target function for an AI agent and to learn from optimizing it. They settled on a single-label classification task, which offers a definitive "yes/no" outcome. For instance, categorizing an item with a true label and a set of available labels allows for a direct comparison: "Is the true label equal to the predicted label?" This provides a clear signal for accuracy calculation. The experiment involved classifying papers from arXiv. The setup included a dataset split into 200 items for fitting, 100 for validation, and 300 for testing to prevent overfitting. The agent was a simple prompt-based classifier using GPT-5-4-nano, chosen for its cost-effectiveness. The optimization process utilized Claude Opus 4.8, which proposed prompt updates based on the GPT-5-4 prompting guide and a task markdown file describing the loop. The results were encouraging. The baseline accuracy started at 68%, and after several iterations, it reached 83%, eventually plateauing around 80%. Notably, the first iteration alone yielded a 10% increase in accuracy. The system learned to add general classification approaches, refine decision-making between similar classes, and incorporate examples of frequently confused labels. This demonstrated that even a simple model, when guided by a clear target function and a structured improvement loop, can achieve significant performance gains. ## Translating High-Signal Feedback to Diverse Applications Schäfer then addressed the challenge of translating this "right/wrong" high-signal feedback to other, less deterministic AI applications. She argued that while metrics like correctness, helpfulness, or hallucination are common, they often represent "low signal" feedback, especially when based on scales (e.g., 0 to 1, 1 to 5). For these metrics to be effective for auto-improvement, each point on the scale needs to be meticulously defined with specific criteria, which is often not the case. Instead, Schäfer advocated for developing "high signal" target functions that translate quality criteria into clear yes/no answers. Examples include: - "Is the answer based on the knowledge base?" (Yes/No) - "Is the brand voice correctly used?" (Yes/No) - "Did we ensure our brand name was written correctly?" (Yes/No) - "Were any known failure modes triggered?" (Categorize out of 5) She stressed that great target functions require sufficient data volume and high-signal evaluators, often developed through collaboration with domain experts. These experts can provide concrete examples, identify nuanced failure modes, and define what "good" truly means in a specific context. ## The Path Forward: Expertise, Data, and Generalization Schäfer outlined a three-step approach to building effective self-improvement systems: 1. **Work with your experts:** Use them to create examples, analyze sample runs, identify failure modes, and define what "good" means. 2. **Review production data as a human:** It's impossible to capture the full scope initially. Scale your dataset with production traces and behaviors to identify typical failure modes. 3. **Design a system that generalizes:** Incorporate validation sets, provide clear instructions for generalization, and include escape hatches to prevent token burning when the system hits a wall. Ultimately, Schäfer concluded, the goal is to create systems where humans and AI agents collaborate on continuous improvement, rather than simply burning tokens on inefficient processes. By focusing on domain expertise and high-signal feedback, developers can build more reliable and performant AI applications. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.