The vast sums locked in cryptocurrency smart contracts are increasingly a target for sophisticated AI. To gauge this evolving threat landscape, researchers have launched EVMbench, a new benchmark suite designed to rigorously evaluate AI agents' prowess in identifying, fixing, and exploiting vulnerabilities within blockchain environments.
Smart contracts, the backbone of decentralized finance, manage over $100 billion in open-source crypto assets. As AI models become more adept at coding, their ability to navigate these complex financial systems, both for malicious and defensive purposes, requires careful measurement. EVMbench aims to provide this critical assessment, fostering the development of AI systems for auditing and fortifying deployed contracts.
EVMbench: A Tri-Modal AI Security Challenge
Developed in collaboration with Paradigm, EVMbench presents AI agents with three distinct challenges:
- Detect: Agents must audit smart contract code to identify known vulnerabilities, with scores based on recall and associated audit rewards.
- Patch: AI must modify vulnerable contracts to eliminate exploits while preserving essential functionality, verified through automated tests and exploit checks.
- Exploit: Agents are tasked with executing full fund-draining attacks against contracts in a secure, sandboxed blockchain environment.
The benchmark draws on 120 curated vulnerabilities sourced from 40 audits, many from open code competitions like Code4rena. It also incorporates scenarios from the Tempo blockchain's security audits, adding relevance for payment-oriented smart contracts.
Creating these task environments involved adapting existing exploit tests and scripts, or manually writing them when necessary. For patching, vulnerabilities were ensured to be fixable without breaking compilation. Exploit modes featured custom graders and red-teaming to prevent agents from cheating the system.