Claude's Corner: Overshoot, Real-Time AI Vision That Moves as Fast as the World

Overshoot is building real-time AI vision infrastructure that connects any video stream to any VLM in under 200ms. Three lines of code. The API is simple. The performance is not. Here's how they do it, and what it takes to replicate it.

Jul 18 at 11:13 AM10 min read

Claude's Corner: Overshoot, Real-Time AI Vision That Moves as Fast as the World

TL;DR

Overshoot connects any video stream to any Vision Language Model in under 200ms. The API is three lines of code; the codec-level frame extraction and edge deployment that make it 10x faster than alternatives is anything but.

6.8

Build difficulty

Real-Time Vision AI Is About to Eat Everything, Overshoot Is the Pick-and-Shovel Play

For the last two years, every AI demo has been the same: upload an image, wait three seconds, get a result. That's fine for generating memes. It's useless for anything that actually moves.

Physical security cameras don't pause for inference. Robots don't wait for your API call to return. A goalkeeper can't hold still while a VLM figures out where the ball is going. Real-time vision AI requires a fundamentally different infrastructure stack, and almost nobody has built it.

Overshoot has. Sub-200ms from stream in to structured JSON out. Any video source. Any VLM. Three lines of code. That's the pitch, and the team has the receipts to back it up.

What Overshoot Actually Does

Overshoot is an API infrastructure layer for real-time vision applications. You connect a video stream, phone camera, webcam, YouTube live, RTSP security feed, screen share, and Overshoot handles everything from frame extraction to VLM inference to returning structured results.

The customer is the developer building the vision app. They don't want to manage FFmpeg pipelines, figure out WebRTC handshakes, or benchmark which VLM gives the fastest response for their specific use case. They want to write overshoot.connect(stream_url, on_result=callback) and move on with their lives.

Business model is usage-based API pricing. You pay per inference run, per frame analyzed, or per minute of stream time, the classic pick-and-shovel approach that scales with your customers without requiring Overshoot to build the end applications themselves.

Three hundred developers are already using it. The use cases are sprawling: physical security (real-time anomaly detection), gaming (AI that watches what's on screen), robotics (perception pipelines), sports analytics (tracking player movements at 60fps), consumer apps (live translation, accessibility tools). The connecting thread is that all of them need fast vision inference at streaming frame rates.

How the Pipeline Actually Works

Video Ingestion: Every Protocol, One Endpoint

The first problem is just getting video into the system cleanly. Overshoot accepts WebRTC (browser and mobile), RTSP (IP cameras, security systems), HLS (live streaming platforms, CDN-delivered content), and direct file upload for near-live processing.

Each protocol has different latency characteristics and buffering behavior. WebRTC is designed for sub-100ms delivery but is notoriously annoying to implement correctly, ICE negotiation, STUN/TURN server management, and codec negotiation all add complexity. RTSP is simpler but comes with higher latency by default. HLS operates on 2-10 second segment windows. Handling all four cleanly without accidentally inheriting the worst latency characteristics of each protocol is non-trivial work.

The Codec Layer: Where the Real Latency Wins Happen

This is where Overshoot earns its performance claims. Most vision API platforms treat video as a container for JPEG frames, decode the video, extract frames, JPEG-compress each one, send to VLM. Straightforward. Also catastrophically inefficient.

H.264 and H.265 video is already spatially compressed. Intelligently, the codec stores only what changed from the previous frame in most cases (P-frames and B-frames). Decoding to raw pixels and re-encoding to JPEG throws away this information and burns CPU cycles you didn't need to spend.

Overshoot's custom codec-level frame extraction works at the compressed bitstream level. The team, with a GPU kernel engineer from Meta AI and an Intel Computer Vision AI Frameworks Engineer, can extract frame data without full decompression where possible, and can make smarter decisions about which frames are worth decoding at all based on the compressed-domain signal alone.

This is not stuff you Google. This is the kind of knowledge that lives in compiler engineering teams at chip companies and distributed systems teams at video platforms. It's why the team matters as much as the product.

Adaptive Frame Sampling: Not Every Frame Needs a PhD

A 30fps video stream is 30 frames per second. Running VLM inference on every frame would be astronomically expensive and completely unnecessary for most applications. A security camera watching an empty parking lot doesn't need GPT-4o Vision 30 times a second.

Overshoot's adaptive sampling layer decides which frames actually need inference. The core signals are scene change detection (did something significant happen visually?), motion magnitude (how much is moving, and where?), and application-specific heuristics (a sports analytics app might care about every frame during a play, but almost nothing between them).

Scene change detection at the codec level is actually cheap, the codec already has motion vectors and residual energy signals baked in. You can compute a meaningful "how much changed?" score without fully decoding the frame. This is exactly the kind of compression-domain trick that turns an expensive inference pipeline into an efficient one.

The result is configurable target inference rates, maybe 2 frames per second for a static scene, 10 for high activity, with the sampling engine handling the decision-making automatically.

VLM Routing: Picking the Right Model for the Job

There is no single best VLM for all vision tasks. GPT-4o Vision gives you high accuracy but slower latency and higher cost. Gemini Flash is faster and cheaper but has different capability tradeoffs. Claude has particular strengths in structured output and reasoning over visual content. Open-source models running on dedicated GPU clusters can be faster still for specific fine-tuned use cases.

Overshoot routes inference requests across models based on configurable criteria: latency budget (is this a real-time alert or a background analysis?), cost tolerance, accuracy requirements, and historical performance on similar frame content. This routing layer also handles retries, fallbacks, and load balancing across API providers.

The operational complexity here is substantial. You're managing API keys, rate limits, error handling, and latency SLAs across multiple external providers simultaneously, while guaranteeing end-to-end latency commitments to your customers.

Edge Deployment: Geographic Proximity Matters

Physics is the ultimate latency constraint. A roundtrip from a camera in Frankfurt to a US East Coast inference server adds 80-120ms before you've done any computation. For sub-200ms total, that's more than half your budget spent on the speed of light.

Overshoot runs edge inference servers distributed geographically, routing incoming streams to the nearest available worker. This isn't just running a few VMs in different regions, it's building an intelligent routing layer that accounts for current server load, VLM availability at each edge, and the latency/cost tradeoffs of serving from a more distant but less loaded edge node.

Combined with the codec optimizations, adaptive sampling, and optimized VLM routing, the result is consistent sub-200ms inference, 10x faster than alternatives that run on centralized infrastructure and treat video like a slow sequence of images.

Difficulty Score

Discipline	Score	Notes
ML / AI	8 / 10	VLM integration itself is API calls, but adaptive sampling, scene change heuristics, and building quality routing logic across models requires real ML intuition. Fine-tuning for latency versus accuracy tradeoffs is genuinely hard.
Data	5 / 10	The data problems are real but standard: high-throughput streaming ingestion, time-series inference logs, usage metering. Nothing exotic, but you need to get the I/O right at volume.
Backend	9 / 10	This is mostly a backend problem. Codec-level video processing, streaming protocol handling, low-latency distributed systems, multi-region routing, and stateful stream management across WebRTC/RTSP/HLS simultaneously. This is the hard part.
Frontend	4 / 10	The product is an API. There's a dashboard, documentation, and probably a stream preview UI, none of it is particularly special. The difficulty lives entirely in the plumbing.
DevOps	8 / 10	Multi-region edge deployment with sub-200ms SLAs is serious infrastructure work. GPU cluster management, streaming server orchestration, auto-scaling on bursty video workloads, and zero-downtime deploys for always-on streams. Kubernetes would be the starting point, not the ending point.

The Moat: What's Genuinely Hard Here

The API surface is not the moat. Anyone can wrap VLM APIs in a streaming endpoint this weekend. That's not the point.

The moat is the combination of codec-level optimization, streaming protocol expertise, GPU kernel tuning, and inference serving architecture that produces consistent sub-200ms latency at scale. Each of those disciplines is independently rare. Having all four in a founding team, Uber pricing systems (distributed, latency-sensitive), Meta GPU kernels, Intel CV frameworks, Intel acquisition founding engineer, is genuinely uncommon.

There's also a data flywheel starting to emerge. As more developers use the platform, Overshoot accumulates signal about which VLMs perform best for which frame types, what sampling rates work for which application categories, and where geographic latency bottlenecks appear. That routing intelligence compounds over time.

The hard technical work is also self-reinforcing. Once you've built custom codec-level extraction that's meaningfully faster than the naive approach, you have a performance gap that competitors can't close without doing the same deep work. It's not a one-way door, but it's a heavy door.

What's easier: building a basic demo that connects a webcam to GPT-4o Vision and calls it a real-time vision platform. That takes a weekend. Plenty of these exist. They don't work at 30fps, they don't have sub-200ms latency, and they fall over at any real load. The demos are easy. The infrastructure is not.

Replicability Score

52out of 100

Here's the honest breakdown of why it lands at 52.

The things you can replicate quickly: the API design, the SDK interface, the VLM routing logic in its basic form, the billing infrastructure, the dashboard, and the documentation. A competent team of two engineers could have a working prototype in a week that passes a demo. It would accept streams. It would return JSON. It would feel like Overshoot.

The things you cannot replicate quickly: the codec-level frame extraction that achieves the latency numbers, the adaptive sampling that makes it cost-efficient at scale, the inference batching that keeps GPU utilization high without blowing latency budgets, and the multi-region edge deployment that handles geographic latency. These require engineers who have done this specific combination of work before.

The expertise required isn't just "good engineers", it's specifically GPU kernel engineers who understand video codecs, and streaming infrastructure engineers who understand inference serving. That's a narrow Venn diagram. You could hire your way into it over 6-12 months. You could not bootstrap your way into it over a weekend.

52 means: if you really wanted to compete, you could. It would take the right 2-3 engineers, a year of focused work, and you'd probably still be chasing Overshoot's performance numbers while they were iterating on the next layer of optimization. Not impossible. Not a weekend project. A real engineering effort against a team with a head start and the exact right expertise.

Why Now

VLMs crossed a capability threshold in 2024-2025 where they became genuinely useful for real-time vision tasks, not just image description, but spatial reasoning, object tracking, action recognition, and anomaly detection. The models exist. The developer demand exists. The infrastructure didn't.

Overshoot is the Cloudflare Workers moment for vision AI, the layer that makes the powerful underlying technology accessible without requiring every developer to become a systems engineer to use it. The timing is right because the models got good enough to be worth building on top of, and the systems engineering gap is still large enough to matter.

Three hundred developers don't adopt an API infrastructure tool because the demo is pretty. They adopt it because it solves a real problem. Overshoot is solving the real problem.

© 2026 StartupHub.ai. All rights reserved. Do not enter, scrape, copy, reproduce, or republish this article in whole or in part. Use as input to AI training, fine-tuning, retrieval-augmented generation, or any machine-learning system is prohibited without written license. Substantially-similar derivative works will be pursued to the fullest extent of applicable copyright, database, and computer-misuse laws. See our terms.

Build This Startup with Claude Code

Complete replication guide — install as a slash command or rules file

# Build a Real-Time Vision API (Overshoot Clone) with Claude Code

A 7-step guide to building your own real-time video inference platform. This covers the full stack: stream ingestion, frame sampling, VLM routing, edge deployment, SDK, and billing. Each step includes what to build, what files to create, and what to ask Claude Code.

---

## Step 1: Project Setup & Database Schema

### Goal
Stand up a PostgreSQL schema that supports multi-tenant stream management, inference tracking, API key auth, and usage metering.

### Database schema

```sql
CREATE TABLE api_keys (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL,
  key_hash TEXT NOT NULL UNIQUE,
  key_prefix TEXT NOT NULL,
  name TEXT,
  scopes TEXT[] DEFAULT '{streams:read,streams:write,inference:run}',
  rate_limit_rpm INTEGER DEFAULT 100,
  is_active BOOLEAN DEFAULT TRUE,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  last_used_at TIMESTAMPTZ
);

CREATE TABLE streams (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL,
  api_key_id UUID REFERENCES api_keys(id),
  source_type TEXT NOT NULL CHECK (source_type IN ('webrtc','rtsp','hls','file')),
  source_url TEXT,
  webrtc_session_id TEXT,
  status TEXT DEFAULT 'pending' CHECK (status IN ('pending','active','paused','ended','error')),
  config JSONB DEFAULT '{}',
  created_at TIMESTAMPTZ DEFAULT NOW(),
  started_at TIMESTAMPTZ,
  ended_at TIMESTAMPTZ,
  metadata JSONB DEFAULT '{}'
);

CREATE TABLE inference_runs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  stream_id UUID NOT NULL REFERENCES streams(id),
  tenant_id UUID NOT NULL,
  frame_timestamp_ms BIGINT NOT NULL,
  frame_index BIGINT,
  vlm_provider TEXT NOT NULL,
  vlm_model TEXT NOT NULL,
  prompt TEXT,
  result JSONB,
  tokens_input INTEGER,
  tokens_output INTEGER,
  latency_ms INTEGER,
  vlm_latency_ms INTEGER,
  sampling_reason TEXT,
  scene_change_score FLOAT,
  motion_score FLOAT,
  status TEXT DEFAULT 'pending',
  error_text TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  completed_at TIMESTAMPTZ
);

CREATE TABLE usage_logs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL,
  period_start TIMESTAMPTZ NOT NULL,
  period_end TIMESTAMPTZ NOT NULL,
  stream_minutes FLOAT DEFAULT 0,
  inference_count INTEGER DEFAULT 0,
  tokens_input BIGINT DEFAULT 0,
  tokens_output BIGINT DEFAULT 0,
  vlm_cost_usd FLOAT DEFAULT 0,
  billed_amount_usd FLOAT DEFAULT 0,
  stripe_usage_record_id TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE(tenant_id, period_start)
);
```

**Ask Claude Code:** "Create a FastAPI project with SQLAlchemy async models matching this schema. Add Alembic migrations, a settings module using pydantic-settings, and a database connection pool with asyncpg. Include a seed script that creates a test tenant and API key."

---

## Step 2: Video Ingestion Layer

### Goal
Accept video streams from WebRTC, RTSP, HLS, and file upload. Extract frames efficiently using FFmpeg.

### Core approach

Use FFmpeg subprocess for RTSP/HLS with `-fflags nobuffer -flags low_delay` to minimize input buffering. For WebRTC, use `aiortc`. Extract raw RGB frames at a configurable target FPS. Run `ffprobe` first to get stream dimensions.

Key FFmpeg flags for low latency:
- `-fflags nobuffer`, reduce input buffering
- `-flags low_delay`, minimize decode delay  
- `-rtsp_transport tcp`, more reliable than UDP
- Output to stdout as rawvideo for zero-copy frame passing

**Ask Claude Code:** "Implement a StreamManager class that accepts a stream config (source_type, source_url, stream_id) and starts the appropriate ingestion pipeline. Expose an async frame queue. Include proper cleanup on stream end and error handling for dropped connections with exponential backoff reconnect."

---

## Step 3: Adaptive Frame Sampling

### Goal
Decide intelligently which frames need VLM inference. Cut costs 80-95% without missing important events.

### Algorithm

1. **Scene change detection**, histogram difference in HSV space (robust to lighting changes). Use `cv2.compareHist` with `HISTCMP_BHATTACHARYYA`. Threshold ~0.15.
2. **Motion detection**, dense optical flow via `cv2.calcOpticalFlowFarneback`. Mean magnitude normalized to 0-1. Threshold ~0.08.
3. **Scheduled fallback**, always infer at least every N seconds (default 2s) regardless of activity.
4. **Minimum interval**, enforce minimum time between inferences to avoid flooding VLM at configured target FPS.

Store scene_change_score and motion_score in `inference_runs` so you can tune thresholds per application.

**Ask Claude Code:** "Wire AdaptiveFrameSampler into StreamManager. Frames where should_infer=True get placed on an inference_queue. Add configurable per-stream sampling params stored in streams.config JSONB. Expose a /streams/{id}/sampling-stats endpoint returning inference rate, skip rate, and top skip reasons over the last hour."

---

## Step 4: VLM Router

### Goal
Route inference requests to the right VLM based on latency budget, cost, and configured preferences. Handle retries and fallbacks.

### Provider selection logic

- Under 200ms budget → Gemini Flash (fastest)
- Higher quality budget → GPT-4o or Claude Sonnet
- Track rolling p50 latency per provider (last 20 calls)
- Fallback chain: OpenAI → Google → Anthropic on failure
- 3 retry attempts with 50ms backoff between attempts

### Cost tracking (per 1M tokens)

| Model | Input | Output |
|-------|-------|--------|
| gpt-4o-mini | $0.15 | $0.60 |
| claude-haiku-4-5 | $0.80 | $4.00 |
| gemini-flash-2-0 | $0.075 | $0.30 |

Apply 3x markup for customer billing. Store vlm_cost_usd and billed_amount in inference_runs.

**Ask Claude Code:** "Add a VLMRouter.benchmark() method that runs a test frame through all three providers and returns latency/cost results. Add a /streams/{id}/router-stats endpoint showing p50/p95 latency per provider and estimated cost per 1000 inference runs."

---

## Step 5: Edge Deployment

### Goal
Deploy inference workers in multiple regions so geographic latency doesn't blow the 200ms budget.

### Fly.io multi-region setup

Deploy workers to: `iad` (US East), `lax` (US West), `fra` (Europe), `nrt` (Japan), `sin` (Singapore).

Route each stream to the nearest healthy worker using geodesic distance from client IP coordinates. Health check every 10 seconds. If a region's p99 latency exceeds 150ms, automatically route to next nearest.

```toml
# fly.toml
[regions]
  iad = { count = 2 }
  lax = { count = 2 }
  fra = { count = 2 }
  nrt = { count = 1 }
  sin = { count = 1 }
```

Dockerfile: `python:3.12-slim` + FFmpeg + OpenCV. Use `uvicorn` with `--workers 4`.

**Ask Claude Code:** "Create a health check loop that pings each region endpoint every 10 seconds and updates the EdgeRouter health map. Implement a fallback chain so that if the nearest region is over 150ms average latency, the router automatically tries the second-nearest."

---

## Step 6: SDK & API

### Goal
Give developers a dead-simple interface: three lines of code to connect a stream.

### WebSocket protocol

Client sends: `{"type": "connect", "stream_url": "...", "config": {"prompt": "...", "sampling_fps": 2.0}}`  
Server responds: `{"type": "connected", "stream_id": "..."}`  
Server pushes: `{"type": "inference_result", "frame_index": N, "timestamp_ms": T, "result": {...}, "latency_ms": 145, "vlm_model": "gemini-flash-2-0"}`

### Python SDK usage (3 lines)
```python
client = Overshoot(api_key="os_live_...")
stream_id = await client.connect(
    stream_url="rtsp://camera.local/stream",
    on_result=lambda r: print(r.result),
    prompt="Alert if a person enters the frame.",
)
```

Publish to PyPI as `overshoot-sdk`. JS equivalent for npm as `overshoot-sdk` using browser WebSockets.

**Ask Claude Code:** "Build the FastAPI WebSocket endpoint at /v1/streams/connect. Add authentication middleware that validates the Authorization header against the api_keys table. Wire it to StreamManager → AdaptiveFrameSampler → VLMRouter and push results back to the client WebSocket."

---

## Step 7: Billing & Metering

### Goal
Usage-based billing via Stripe Metered Subscriptions. Count tokens per frame, enforce quotas, aggregate hourly.

### Stripe setup
1. Create Product: "Overshoot API"
2. Create metered Price with `usage_type: metered` and `aggregate_usage: sum`
3. On tenant signup, create Stripe Customer + Subscription
4. Store `stripe_subscription_item_id` on tenant record

### Metering pattern
- After each inference_run, upsert into `usage_logs` hourly bucket using `ON CONFLICT DO UPDATE`
- Background job (APScheduler, hourly) calls `stripe.SubscriptionItem.create_usage_record()` for each unbilled hour
- Mark billed rows with `stripe_usage_record_id`

### Quota enforcement
Check current month spend from `usage_logs` before allowing new stream connections. Return HTTP 429 with remaining quota in headers.

**Ask Claude Code:** "Add quota enforcement middleware to FastAPI that checks monthly spend before allowing new stream connections. Create an APScheduler background task for hourly Stripe reporting. Add /billing/usage endpoint returning current month spend, quota remaining, and a breakdown by day and VLM provider."

---

## Full Stack Summary

| Layer | Tech | Key Challenge |
|-------|------|---------------|
| Ingestion | FFmpeg, aiortc | Low-latency protocol handling |
| Frame sampling | OpenCV, NumPy | Balancing cost vs. coverage |
| VLM routing | OpenAI/Anthropic/Google SDKs | p99 latency under 200ms |
| Edge deployment | Fly.io multi-region | Geographic routing, failover |
| SDK | Python asyncio, WebSockets | Clean developer interface |
| Billing | Stripe Metered | Accurate per-inference counting |
| Database | PostgreSQL + asyncpg | High-throughput write performance |

The gap between a working prototype and Overshoot's performance is entirely in the codec layer (Step 2), inference batching within the VLM router (Step 4), and multi-region edge routing (Step 5). Those three pieces require specialists. Everything else is achievable by a solid generalist engineer in a week.

Install for:

claude-code-skills.md

#YC W2026 #computer vision #real-time AI #video inference #developer tools #AI infrastructure #VLM #WebRTC #edge computing