Claude's Corner: Pocket — The AI Hardware Startup That Quietly Hit $27M ARR

---
description: Build a Pocket clone — an ambient conversation recorder that turns audio into transcripts, summaries, and action items
---

# Build Pocket: Ambient AI Conversation Intelligence

## What You're Building
A web application that accepts audio recordings (from a phone, microphone, or uploaded file), transcribes them with speaker diarization, and uses an LLM to produce structured summaries, action items, and a mind map of topics. This is the entire Pocket software layer — minus the hardware.

## Tech Stack
- **Frontend:** Next.js 14 (App Router) + shadcn/ui
- **Backend:** Next.js API routes + Python FastAPI microservice for AI pipeline
- **Database:** Supabase (Postgres + Storage for audio files)
- **AI/ML:** Whisper (OpenAI API) for transcription, pyannote.audio for diarization, Claude or GPT-4o for summarization
- **Key Libraries:** openai, anthropic, pyannote.audio, ffmpeg-python

## Step 1: Project Setup

```bash
npx create-next-app@latest pocket-clone --typescript --tailwind --app
cd pocket-clone
npx shadcn@latest init
npm install @supabase/supabase-js openai @anthropic-ai/sdk
```

Supabase SQL:
```sql
CREATE TABLE recordings (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID REFERENCES auth.users,
  title TEXT,
  audio_url TEXT NOT NULL,
  status TEXT DEFAULT 'pending',
  transcript TEXT,
  summary TEXT,
  action_items JSONB DEFAULT '[]',
  mind_map JSONB DEFAULT '{}',
  created_at TIMESTAMPTZ DEFAULT NOW()
);
```

## Step 2: Core Data Models

```typescript
export interface ActionItem {
  text: string;
  owner?: string;
  completed: boolean;
}

export interface Recording {
  id: string;
  title: string;
  status: 'pending' | 'processing' | 'done' | 'error';
  transcript?: string;
  summary?: string;
  actionItems: ActionItem[];
  mindMap: { label: string; children: unknown[] };
}
```

## Step 3: Audio Upload + Storage

```typescript
// app/api/recordings/upload/route.ts
export async function POST(req: Request) {
  const formData = await req.formData();
  const file = formData.get('audio') as File;
  const userId = formData.get('userId') as string;
  const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_KEY!);

  const fileName = `${userId}/${Date.now()}-${file.name}`;
  await supabase.storage.from('audio-recordings').upload(fileName, file);
  const { data: { signedUrl } } = await supabase.storage
    .from('audio-recordings').createSignedUrl(fileName, 86400);

  const { data: recording } = await supabase
    .from('recordings')
    .insert({ user_id: userId, audio_url: signedUrl, status: 'pending' })
    .select().single();

  await fetch(`${process.env.PIPELINE_URL}/process`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ recording_id: recording.id, audio_url: signedUrl })
  });

  return Response.json({ recordingId: recording.id });
}
```

## Step 4: AI Processing Pipeline (Python FastAPI)

```python
from fastapi import FastAPI, BackgroundTasks
from openai import OpenAI
from anthropic import Anthropic
import httpx, json, os

app = FastAPI()
openai_client = OpenAI()
anthropic_client = Anthropic()

@app.post("/process")
async def process(payload: dict, background_tasks: BackgroundTasks):
    background_tasks.add_task(run_pipeline, payload["recording_id"], payload["audio_url"])
    return {"status": "queued"}

async def run_pipeline(recording_id: str, audio_url: str):
    async with httpx.AsyncClient() as client:
        audio_bytes = (await client.get(audio_url)).content

    with open("/tmp/audio.mp3", "wb") as f:
        f.write(audio_bytes)

    with open("/tmp/audio.mp3", "rb") as f:
        transcription = openai_client.audio.transcriptions.create(
            model="whisper-1", file=f, response_format="verbose_json")

    transcript_text = transcription.text

    response = anthropic_client.messages.create(
        model="claude-sonnet-4-5", max_tokens=2048,
        system="""Extract from transcript:
1. Summary (3-5 sentences)
2. Action items as JSON: [{text, owner, due}]
3. Mind map as JSON: {label, children: [{label, children}]}
Respond ONLY with valid JSON: {summary, actionItems, mindMap}""",
        messages=[{"role": "user", "content": f"Transcript:\n\n{transcript_text}"}])

    extracted = json.loads(response.content[0].text)

    async with httpx.AsyncClient() as client:
        await client.patch(
            f"{os.environ['SUPABASE_URL']}/rest/v1/recordings?id=eq.{recording_id}",
            headers={"apikey": os.environ["SUPABASE_SERVICE_KEY"],
                     "Authorization": f"Bearer {os.environ['SUPABASE_SERVICE_KEY']}"},
            json={"status": "done", "transcript": transcript_text,
                  "summary": extracted["summary"],
                  "action_items": extracted["actionItems"],
                  "mind_map": extracted["mindMap"]})
```

## Step 5: Real-time Results UI

```typescript
// app/recordings/[id]/page.tsx
"use client";
export default function RecordingPage({ params }: { params: { id: string } }) {
  const [recording, setRecording] = useState(null);
  const supabase = createClient(SUPABASE_URL, SUPABASE_ANON_KEY);

  useEffect(() => {
    supabase.from('recordings').select('*').eq('id', params.id).single()
      .then(({ data }) => setRecording(data));

    const channel = supabase.channel('rec-' + params.id)
      .on('postgres_changes',
        { event: 'UPDATE', schema: 'public', table: 'recordings', filter: `id=eq.${params.id}` },
        (payload) => setRecording(payload.new))
      .subscribe();

    return () => { supabase.removeChannel(channel); };
  }, []);

  return (
    <div className="max-w-3xl mx-auto p-6 space-y-6">
      <h1 className="text-2xl font-bold">{recording?.title}</h1>
      <div className="rounded-xl border p-5 bg-card">
        <h2 className="font-semibold mb-2">Summary</h2>
        <p className="text-muted-foreground">{recording?.summary}</p>
      </div>
      <div className="rounded-xl border p-5 bg-card">
        <h2 className="font-semibold mb-3">Action Items</h2>
        {recording?.action_items?.map((item, i) => (
          <div key={i} className="flex gap-2 py-1">
            <input type="checkbox" />
            <span>{item.text}</span>
          </div>
        ))}
      </div>
    </div>
  );
}
```

## Step 6: Frontend Pages

- `/` — Landing page with upload CTA
- `/dashboard` — Recording list with status badges
- `/recordings/[id]` — Summary, action items, transcript, mind map
- `/upload` — Drag-and-drop (accept .mp3, .m4a, .wav, .ogg)

shadcn components: Card, Badge, Progress, Tabs, Button, Dialog.

## Step 7: Deploy

Frontend to Vercel. Python pipeline to Railway or Render with this Dockerfile:

```dockerfile
FROM python:3.11-slim
RUN apt-get update && apt-get install -y ffmpeg
RUN pip install fastapi uvicorn openai anthropic httpx python-multipart
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```

## Key Insights
- Whisper verbose_json gives segment timestamps — use for audio playback sync
- Whisper does NOT do speaker diarization — add pyannote.audio or AssemblyAI Speaker Labels
- Supabase Realtime for live status updates beats polling every time
- Process audio in background tasks — never block on a 30-min file
- Claude structured JSON output is more consistent than GPT-4o for action item schemas

## Gotchas
- Next.js API routes cap at 4MB — disable bodyParser and stream to Supabase Storage directly
- Normalize audio to WAV/MP3 with ffmpeg before Whisper (browser recordings vary by format)
- Whisper has a 25MB file size limit — chunk large files with pydub first
- Cost estimate: Whisper = $0.006/min + Claude Sonnet ~$0.003/1K tokens. A 30-min meeting costs ~$0.20 to process