#LLM Evaluation
8 articles with this tag
Meta's Nishant Gupta on Evaluating Agentic AI Systems
Nishant Gupta from Meta's Superintelligence Labs discusses the shift from accuracy-based evaluation to reliability-focused methods for agentic AI systems.
ClinEnv: Bridging LLM Gaps in Clinical Decision-Making
The ClinEnv benchmark reveals LLMs struggle with sequential medical decision-making, showing a gap between diagnostic and management capabilities.
LinkedIn Tries Real-World AI Benchmarking
LinkedIn's new Crosscheck platform aims to provide real-world AI model performance insights tailored to professional roles and tasks.
DeepWeb-Bench: Beyond Frontier LLM Claims
DeepWeb-Bench benchmark exposes derivation and calibration as major LLM failure points, revealing domain specialization and the inadequacy of current evaluations.
LLM Drift: A Structural Blind Spot
LLMs suffer from structural temporal drift, rendering them confidently outdated. A new geometric probe detects this, outperforming standard methods.
LLMs Fail Esoteric Code Tasks
Frontier LLMs show a dramatic capability gap on a new benchmark using esoteric programming languages, revealing a reliance on memorization over reasoning.
Balyasny's AI Engine
Balyasny Asset Management built a powerful AI research engine using OpenAI models, slashing analysis times and boosting investment team confidence.
Context-Aware Guardrails Tested
Mozilla.ai tested context-aware guardrails for LLMs in a humanitarian context, revealing crucial multilingual performance disparities and the need for robust, domain-specific safety policies.