Cisco Vets Bank AI Deploys in Weeks, Not a Year

Al Chen, who leads AI platform work at Cisco, told AI Engineer his team now pushes AI observability into five major banks in weeks instead of 12 months by making compliance show its work.

The delay is rarely model quality. Chen said most risk teams act like his 10th grade math teacher Mr. Huff, who graded the proof more than the answer to four decimal places. For production workloads that millions of customers touch in mobile apps and web logins, banks block outbound egress, require air-gapped on-prem installs, and demand evidence, not assurances, for every image pulled into their clusters.

Pillar one is CVEs. Chen said each platform ships multiple images, each with multiple common vulnerabilities and exposures, and banks run different scanners like Sysdig, Wiz and Prisma with their own severity levels and SLAs. Transient dependencies from open-source projects often cannot be remediated until maintainers ship a fix, and one bank may want package 3.8 while another rejects it for having more CVEs. Bumping a version does not prove the exploit is gone, and scanners only check the version string.

Chen's fix is a Harbor task. Give an agent the CVE, the environment with the actual Docker images, and instructions to write an exploit against the old package version, then test whether 2.6.2 still fails. Run it across Docker, Modal or Daytona sandboxes in parallel and keep the passing tests as evidence. It is thorough but still probabilistic, and Chen acknowledged his team must constantly mine release notes to explain to bank security why some CVEs remain unfixable.

Pillar two is disaster recovery, and Chen singled out ClickHouse. Unlike Redis or Postgres, he said, ClickHouse offers no managed failover across regions for data replication, even in its own resiliency docs. His pattern uses ClickHouse in a primary region, a job that backs up to a bucket via the Altinity-maintained ClickHouse Backup project, and a restore job in a DR region with in-region replicas. Backup success logs are not enough, he warned, since a bad disk, bad migration or replication lag can leave row counts divergent, so teams must verify row counts and replica convergence. Replicating to a bucket also forces a data residency choice, and his team often replicates only IDs and trace metadata cross-region while keeping core trace data in one jurisdiction.

Pillar three is leakage through evaluation. Redaction at the SDK, gateway or middleware may clean the UI, but Chen said judges leak. A seemingly innocuous LLM-as-a-judge prompt that asks for a score and explanation so a human can verify accuracy can un-redact PII or MNPI in its reasoning, even when the UI shows redacted fields. His mitigation keeps raw traces for the judge, persists the numeric score, and gates the reasoning behind a check for sensitive data, suppressing it if present. Banks that insist on seeing reasoning get a second judge run on redacted input at the cost of another LLM call per request.

The timeline signals are organizational, not technical. Chen said having an approved image pull process and sourced GPUs shortens delivery, while formal security reviews, custom LLM gateway or RBAC work adds months, and lacking any pull approval or GPU procurement plan pushes it furthest. Against that backdrop, even single-purpose automation is still news elsewhere: Vertafore said it launched Velocity Policy Intake and Policy Check AI agents to automate policy data entry and coverage verification for independent agencies. Shipping bank AI faster is possible, Chen argued, but only if vendors prove the exploit is dead, the data is correct, and the reasoning is clean, and that proof costs extra compute and process every time.