Auditing LLM Agent Skill Integrity

The expansion of LLM agents into real-world applications hinges on their ability to safely leverage privileged third-party capabilities. However, a critical vulnerability exists in the unverified nature of these 'skill artifacts.' The Behavioral Integrity Verification (BIV) framework addresses this gap by formalizing the problem as a typed set comparison between declared and actual capabilities.

The Pervasive Description-Implementation Chasm

Analysis of 49,943 skills from the OpenClaw registry reveals a stark reality: 80.0% of skills deviate from their declared behavior. This pervasive gap, far from being an edge case, surfaces four novel compound-threat categories, highlighting a fundamental challenge in ensuring LLM agent behavioral integrity at scale. The framework instantiates a comparison by pairing deterministic code analysis with LLM-assisted capability extraction to generate structured evidence.

Root Causes: Oversight Dominates Adversarial Intent

The BIV framework not only identifies deviations but also classifies their root causes. The findings indicate that the majority of these discrepancies stem from developer oversight (81.1%), rather than deliberate malicious intent (18.9%). However, the potential for harm is significant, with 5.0% of skills predicted to enable multi-stage attack chains. This underscores the necessity of robust auditing mechanisms to maintain LLM agent behavioral integrity.

BIV: A Scalable Solution for Malicious Skill Detection

Demonstrating its efficacy, the BIV framework achieves a remarkable F1 score of 0.946 on a 906-skill malicious-skill detection benchmark. This performance significantly surpasses existing rule-based and single-pass LLM baselines, offering a scalable and accurate method for auditing agent skills and mitigating associated risks.