Claude's Corner: Cardboard — The Agentic Video Editor That Edits Like a Human

---
description: Build a Cardboard clone — agentic browser-based video editor with AI-driven timeline assembly
---

# Build Cardboard: Agentic Video Editor

## What You're Building
A browser-based video editor where users upload raw footage, describe what they want in plain English, and receive an editable multi-track timeline assembled by an AI agent. The core insight: the AI generates timeline editing *decisions*, not generated video — keeping output trustworthy and editable.

## Tech Stack
- **Frontend:** Next.js 14 (App Router), React, Tailwind CSS
- **Video Rendering:** WebCodecs API + WebGL2 (client-side, no server rendering)
- **Backend:** Next.js API routes, Node.js
- **Database:** Supabase (Postgres + Storage)
- **AI/ML:** Anthropic Claude API (agent orchestration), OpenAI GPT-4o Vision (scene analysis), Whisper (transcription)
- **Auth:** Clerk
- **Key Libraries:** @ffmpeg/ffmpeg (WASM fallback for codecs), mp4box.js (container demuxing), wavesurfer.js (audio waveform)

## Step 1: Project Setup

```bash
npx create-next-app@latest cardboard-clone --typescript --tailwind --app
cd cardboard-clone
npm install @clerk/nextjs @anthropic-ai/sdk openai @supabase/supabase-js
npm install wavesurfer.js uuid zod mp4box
npm install -D @types/uuid
```

File structure:
```
app/
  api/
    analyze-footage/route.ts    # VLM scene analysis pipeline
    assemble-timeline/route.ts  # Claude agent for editing decisions
    transcribe/route.ts         # Whisper transcription
  editor/
    [projectId]/page.tsx        # Main editor UI
  upload/page.tsx
src/
  components/
    Timeline/Track.tsx
    Timeline/Clip.tsx
    Timeline/PlayheadScrubber.tsx
    VideoPreview/WebCodecsPlayer.tsx
    Editor/CommandBar.tsx
    Editor/MediaLibrary.tsx
  lib/
    video/keyframe-extractor.ts
    video/timeline-assembler.ts
    claude-agent.ts
  types/
    timeline.ts
```

## Step 2: Core Data Models

```typescript
// src/types/timeline.ts
export interface MediaClip {
  id: string;
  projectId: string;
  filename: string;
  storageUrl: string;
  duration: number;
  scenes: Scene[];
  transcript?: string;
  keyframes: Keyframe[];
}

export interface Scene {
  startTime: number;
  endTime: number;
  description: string;
  subjects: string[];
  energy: 'low' | 'medium' | 'high';
  isSilent: boolean;
}

export interface Timeline {
  tracks: Track[];
  duration: number;
}

export interface Track {
  id: string;
  type: 'video' | 'audio' | 'caption';
  clips: TimelineClip[];
}

export interface TimelineClip {
  id: string;
  sourceClipId: string;
  startTime: number;   // position on output timeline
  endTime: number;
  sourceStart: number; // in-point from source
  sourceEnd: number;   // out-point from source
}
```

Supabase SQL schema:
```sql
CREATE TABLE projects (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id TEXT NOT NULL,
  name TEXT NOT NULL,
  fps INTEGER DEFAULT 30,
  width INTEGER DEFAULT 1920,
  height INTEGER DEFAULT 1080,
  timeline JSONB DEFAULT '{"tracks": [], "duration": 0}'::jsonb,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE media_clips (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  project_id UUID REFERENCES projects(id) ON DELETE CASCADE,
  filename TEXT NOT NULL,
  storage_path TEXT NOT NULL,
  duration FLOAT,
  scenes JSONB DEFAULT '[]'::jsonb,
  keyframes JSONB DEFAULT '[]'::jsonb,
  transcript TEXT,
  analysis_status TEXT DEFAULT 'pending',
  created_at TIMESTAMPTZ DEFAULT NOW()
);
```

## Step 3: Keyframe Extraction and VLM Analysis

When a clip is uploaded, extract keyframes at 2-second intervals and run each through GPT-4o Vision to build semantic scene understanding.

```typescript
// app/api/analyze-footage/route.ts
import OpenAI from 'openai';
import { createClient } from '@supabase/supabase-js';

const openai = new OpenAI();

export async function POST(req: Request) {
  const { clipId, keyframes } = await req.json();
  // keyframes: { timestamp: number; storageUrl: string }[]

  const scenes = [];

  for (const frame of keyframes) {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{
        role: 'user',
        content: [
          { type: 'image_url', image_url: { url: frame.storageUrl } },
          {
            type: 'text',
            text: 'Analyze this video frame at ' + frame.timestamp + 's. Return JSON: {"description": "one sentence", "subjects": ["list"], "energy": "low|medium|high", "isSilent": boolean, "mood": "string"}'
          }
        ]
      }],
      response_format: { type: 'json_object' }
    });

    const analysis = JSON.parse(response.choices[0].message.content);
    scenes.push({ startTime: frame.timestamp, endTime: frame.timestamp + 2, ...analysis });
  }

  const supabase = createClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL,
    process.env.SUPABASE_SERVICE_ROLE_KEY
  );

  await supabase.from('media_clips')
    .update({ scenes, analysis_status: 'complete' })
    .eq('id', clipId);

  return Response.json({ scenes });
}
```

## Step 4: Claude Agent for Timeline Assembly

The editorial brain — takes natural language and analyzed footage, returns structured timeline operations.

```typescript
// app/api/assemble-timeline/route.ts
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

export async function POST(req: Request) {
  const { prompt, clips, targetDuration } = await req.json();

  const footageContext = clips.map((clip) => ({
    id: clip.id,
    filename: clip.filename,
    totalDuration: clip.duration,
    scenes: clip.scenes.map(s => ({
      time: s.startTime + 's-' + s.endTime + 's',
      description: s.description,
      energy: s.energy,
      subjects: s.subjects,
      isSilent: s.isSilent
    })),
    transcript: clip.transcript?.slice(0, 800)
  }));

  const response = await anthropic.messages.create({
    model: 'claude-opus-4-5',
    max_tokens: 4096,
    system: `You are a professional video editor. Given footage analysis and a user request, return a JSON timeline of editing decisions.
Rules: only use clips from the provided list; sourceStart/sourceEnd must be within clip duration; remove silent sections unless asked to keep them; prefer high-energy scenes for montages; maintain speech flow for talking head content.`,
    messages: [{
      role: 'user',
      content: 'FOOTAGE:
' + JSON.stringify(footageContext, null, 2) + '

REQUEST: "' + prompt + '"
TARGET DURATION: ' + targetDuration + ' seconds

Return JSON with tracks array containing video and audio tracks with clips array. Include a "reasoning" field explaining editorial choices.'
    }]
  });

  const timeline = JSON.parse(response.content[0].text);
  return Response.json(timeline);
}
```

## Step 5: WebCodecs Playback Engine

The hardest part. WebCodecs requires manual container demuxing — mp4box.js extracts encoded chunks, VideoDecoder decodes them, WebGL2 composites tracks.

```typescript
// src/lib/video/webcodecs-player.ts
export class WebCodecsPlayer {
  private decoder: VideoDecoder | null = null;
  private canvas: HTMLCanvasElement;
  private ctx: CanvasRenderingContext2D;

  constructor(canvas: HTMLCanvasElement) {
    this.canvas = canvas;
    this.ctx = canvas.getContext('2d')!;
  }

  async init(videoFile: File) {
    // Feature detect first — always
    if (!('VideoDecoder' in window)) {
      console.warn('WebCodecs not supported, using ffmpeg.wasm fallback');
      return this.initWasmFallback(videoFile);
    }

    this.decoder = new VideoDecoder({
      output: (frame) => {
        this.ctx.drawImage(frame, 0, 0, this.canvas.width, this.canvas.height);
        frame.close(); // CRITICAL: always close frames to avoid memory leaks
      },
      error: (e) => console.error('Decoder error:', e)
    });

    // Use mp4box.js to demux — this is the part most tutorials skip
    // mp4box parses the container and gives you encoded chunks (EncodedVideoChunk)
    // which you feed to VideoDecoder.decode()
    // See: https://gpac.github.io/mp4box.js/
  }

  private async initWasmFallback(videoFile: File) {
    // @ffmpeg/ffmpeg runs ffmpeg in WASM for browsers without WebCodecs
    const { FFmpeg } = await import('@ffmpeg/ffmpeg');
    const ffmpeg = new FFmpeg();
    await ffmpeg.load();
    // Extract frames via: ffmpeg -i input -vf fps=30 frame_%04d.jpg
  }
}
```

## Step 6: Command Bar UI

```tsx
// src/components/Editor/CommandBar.tsx
'use client';
import { useState } from 'react';
import { Button } from '@/src/components/ui/button';
import { Textarea } from '@/src/components/ui/textarea';

const EXAMPLES = [
  'Make a 60s product launch video with upbeat energy',
  'Cut a 30s testimonial highlighting the key benefit',
  'Create 3 x 20s social clips synced to the music',
];

export function CommandBar({ onAssemble, isLoading }) {
  const [prompt, setPrompt] = useState('');
  const [duration, setDuration] = useState(60);

  return (
    <div className="border border-border rounded-lg p-4 bg-card space-y-3">
      <Textarea value={prompt} onChange={e => setPrompt(e.target.value)}
        placeholder="Describe the video you want..." className="min-h-[80px] resize-none" />
      <div className="flex items-center gap-3">
        <label className="text-sm text-muted-foreground">Duration:</label>
        <input type="number" value={duration} onChange={e => setDuration(Number(e.target.value))}
          className="w-20 border border-input rounded px-2 py-1 text-sm bg-background" />
        <span className="text-sm text-muted-foreground">seconds</span>
        <Button onClick={() => onAssemble(prompt, duration)} disabled={!prompt || isLoading} className="ml-auto">
          {isLoading ? 'Assembling...' : 'Assemble Edit'}
        </Button>
      </div>
      <div className="flex flex-wrap gap-2">
        {EXAMPLES.map(ex => (
          <button key={ex} onClick={() => setPrompt(ex)}
            className="text-xs px-2 py-1 rounded bg-muted text-muted-foreground hover:bg-accent transition-colors">
            {ex}
          </button>
        ))}
      </div>
    </div>
  );
}
```

## Step 7: Deploy

```bash
# Vercel is the natural fit for Next.js
vercel

# Required environment variables:
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
NEXT_PUBLIC_SUPABASE_URL=https://xxx.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=...
SUPABASE_SERVICE_ROLE_KEY=...
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_...
CLERK_SECRET_KEY=sk_...

# Supabase Storage: create two buckets
# "footage" — private, increase per-file limit to 10GB in project settings
# "keyframes" — public (these URLs get sent to VLMs for analysis)
```

## Key Insights

- **Timeline-as-output is the right abstraction.** The AI generates editing decisions, not pixels. This keeps output editable, keeps brand voice intact, and avoids hallucination artifacts in final video.
- **Client-side rendering is a real moat.** WebCodecs + WebGL2 in the browser is genuinely hard to get right. Most competitors use server-side rendering which is 10x slower and more expensive. Invest in getting WebCodecs right.
- **VLM keyframe analysis is the foundation.** Denser extraction (every 1s vs every 3s) and better prompts produce dramatically better editorial results. This is where to invest quality engineering time.
- **Silence detection before VLMs saves money.** Run audio RMS energy analysis first to identify boring sections. Skip those frames entirely when sending to GPT-4o Vision. Cuts costs by 40-60%.

## Gotchas

- **WebCodecs needs mp4box.js.** WebCodecs decodes video chunks but does not parse containers. You must demux the MP4/MOV file with mp4box.js to extract EncodedVideoChunks before calling VideoDecoder.decode(). This is the step most tutorials skip.
- **Always close VideoFrames.** Call frame.close() immediately after drawing. Leaked VideoFrames cause GPU memory exhaustion within minutes of playback.
- **Browser support.** Always feature-detect: if (!('VideoDecoder' in window)). Safari support arrived late and still has quirks with certain codecs. H.264/AVC is safest for cross-browser compat.
- **VLM costs scale fast.** GPT-4o Vision at 2-second intervals on a 5-minute clip is roughly 150 frames at ~$0.002/frame = ~$0.30 per upload. Run analysis once, cache aggressively, never re-analyze.
- **Large upload resumability.** Use Supabase Storage TUS chunked upload for files over 50MB. A network drop mid-upload on a 2GB ProRes file without resumability creates catastrophic UX.