#LLM Evaluation
9 articles with this tag
Alejandro Vidal on Rethinking LLM Evaluation with Psychometrics
Alejandro Vidal of Mindmakers advocates for integrating psychometrics into LLM evaluation to move beyond simplistic accuracy scores and gain deeper insights into model intelligence and benchmark quality.
Meta's Nishant Gupta on Evaluating Agentic AI Systems
Nishant Gupta from Meta's Superintelligence Labs discusses the shift from accuracy-based evaluation to reliability-focused methods for agentic AI systems.
ClinEnv: Bridging LLM Gaps in Clinical Decision-Making
The ClinEnv benchmark reveals LLMs struggle with sequential medical decision-making, showing a gap between diagnostic and management capabilities.
LinkedIn Tries Real-World AI Benchmarking
LinkedIn's new Crosscheck platform aims to provide real-world AI model performance insights tailored to professional roles and tasks.
DeepWeb-Bench: Beyond Frontier LLM Claims
DeepWeb-Bench benchmark exposes derivation and calibration as major LLM failure points, revealing domain specialization and the inadequacy of current evaluations.
LLM Drift: A Structural Blind Spot
LLMs suffer from structural temporal drift, rendering them confidently outdated. A new geometric probe detects this, outperforming standard methods.
LLMs Fail Esoteric Code Tasks
Frontier LLMs show a dramatic capability gap on a new benchmark using esoteric programming languages, revealing a reliance on memorization over reasoning.
Balyasny's AI Engine
Balyasny Asset Management built a powerful AI research engine using OpenAI models, slashing analysis times and boosting investment team confidence.
Context-Aware Guardrails Tested
Mozilla.ai tested context-aware guardrails for LLMs in a humanitarian context, revealing crucial multilingual performance disparities and the need for robust, domain-specific safety policies.