Claude's Corner: Stably AI — AI That Writes and Heals Your Tests

---
description: Build a Stably AI clone — AI-powered Playwright test generation and self-healing CI tool
---

# Build Stably AI: AI That Writes and Heals Your E2E Tests

## What You're Building
An AI-powered QA platform that takes plain English test descriptions, generates runnable Playwright test code, executes tests in CI on every PR, and automatically heals broken selectors/assertions using an LLM agent when your UI changes.

## Tech Stack
- **Frontend:** Next.js 14 (App Router), React, Tailwind CSS, shadcn/ui
- **Backend:** Node.js, Express or Next.js API routes
- **Database:** Supabase (PostgreSQL) — stores test suites, run history, healing logs
- **Test Runner:** @playwright/test with custom reporter
- **AI/ML:** Anthropic Claude API (claude-sonnet-4-5 for healing agent, claude-haiku for generation)
- **CI Integration:** GitHub App + GitHub Actions
- **Infrastructure:** Vercel (frontend/API), AWS ECS Fargate (test execution containers)
- **Key Libraries:** @playwright/test, @anthropic-ai/sdk, @octokit/app, cheerio, zod

## Step 1: Project Setup

```bash
npx create-next-app@latest stably-clone --typescript --tailwind --app
cd stably-clone
npm install @playwright/test @anthropic-ai/sdk @octokit/app zod cheerio uuid
npx playwright install chromium
npx shadcn@latest init
```

Directory structure:
```
stably-clone/
  app/
    api/
      generate-tests/     # POST: natural language -> Playwright code
      run-tests/          # POST: execute a test suite
      heal-tests/         # POST: fix broken tests
      github/webhook/     # GitHub App webhook handler
    dashboard/            # Test suite management UI
  lib/
    playwright-runner.ts  # Test execution engine
    test-generator.ts     # LLM -> Playwright code
    healing-agent.ts      # Self-heal logic
    github-app.ts         # GitHub App client
  supabase/
    schema.sql
```

## Step 2: Core Data Models

```sql
CREATE TABLE test_suites (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  org_id UUID NOT NULL,
  repo_full_name TEXT NOT NULL,
  name TEXT NOT NULL,
  base_url TEXT NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE test_cases (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  suite_id UUID REFERENCES test_suites(id) ON DELETE CASCADE,
  title TEXT NOT NULL,
  description TEXT NOT NULL,
  playwright_code TEXT NOT NULL,
  status TEXT DEFAULT 'active',
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE test_runs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  suite_id UUID REFERENCES test_suites(id),
  triggered_by TEXT,
  pr_number INT,
  commit_sha TEXT,
  status TEXT,
  results JSONB,
  artifacts_url TEXT,
  started_at TIMESTAMPTZ DEFAULT NOW(),
  completed_at TIMESTAMPTZ
);

CREATE TABLE healing_events (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  test_case_id UUID REFERENCES test_cases(id),
  run_id UUID REFERENCES test_runs(id),
  original_code TEXT,
  healed_code TEXT,
  failure_reason TEXT,
  healing_notes TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);
```

## Step 3: Test Generation (Natural Language -> Playwright)

```typescript
// lib/test-generator.ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

export async function generatePlaywrightTest(
  description: string,
  baseUrl: string
): Promise<string> {
  const systemPrompt = `You are an expert Playwright test engineer.
Generate complete, runnable Playwright TypeScript test code.

CRITICAL RULES:
1. Use page.getByRole(), page.getByText(), page.getByLabel() as primary locators
2. Add a .describe("intent here") annotation to every locator for self-healing
3. Use explicit await and proper async patterns
4. Include realistic waits: await page.waitForLoadState("networkidle")
5. Output ONLY the TypeScript code, no markdown fences`;

  const message = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 2048,
    messages: [{ role: "user", content: `Base URL: ${baseUrl}\nTest: ${description}` }],
    system: systemPrompt,
  });

  const content = message.content[0];
  if (content.type !== "text") throw new Error("Unexpected response type");
  return content.text;
}
```

## Step 4: Self-Healing Agent

```typescript
// lib/healing-agent.ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

export async function healBrokenTest(
  failedTestCode: string,
  errorMessage: string,
  currentDOMSnapshot: string,
  pageUrl: string
): Promise<{ healed_code: string; notes: string; confidence: string }> {
  const prompt = `You are a Playwright test healing agent.

A test is failing. Update the test code to fix the failure WITHOUT changing what the test is testing.

Failed Test:
${failedTestCode}

Error:
${errorMessage}

Current DOM (relevant elements):
${currentDOMSnapshot.slice(0, 8000)}

Page URL: ${pageUrl}

Find the failing locator using its .describe() intent annotation, locate the correct element in the DOM, and fix the locator.
Return JSON: { "healed_code": "...", "notes": "...", "confidence": "high|medium|low" }`;

  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 4096,
    messages: [{ role: "user", content: prompt }],
  });

  const text = response.content[0].type === "text" ? response.content[0].text : "";
  const jsonMatch = text.match(/\{[\s\S]*\}/);
  if (!jsonMatch) throw new Error("Healing agent returned invalid JSON");
  return JSON.parse(jsonMatch[0]);
}

export async function getDOMSnapshot(page: import("@playwright/test").Page): Promise<string> {
  return page.evaluate(() => {
    const elements = document.querySelectorAll("button, a, input, select, textarea, [role], h1, h2, h3, label");
    return Array.from(elements).map((el) => el.outerHTML).join("\n");
  });
}
```

## Step 5: GitHub App Webhook Handler

```typescript
// app/api/github/webhook/route.ts
import { App } from "@octokit/app";

const app = new App({
  appId: process.env.GITHUB_APP_ID!,
  privateKey: process.env.GITHUB_APP_PRIVATE_KEY!,
  webhooks: { secret: process.env.GITHUB_WEBHOOK_SECRET! },
});

app.webhooks.on("pull_request.opened", async ({ payload }) => {
  const repo = payload.repository.full_name;
  const sha = payload.pull_request.head.sha;
  const prNumber = payload.pull_request.number;
  await queueTestRun(repo, prNumber, sha);
});

export async function POST(request: Request) {
  const body = await request.text();
  const signature = request.headers.get("x-hub-signature-256") ?? "";
  await app.webhooks.verifyAndReceive({
    id: request.headers.get("x-github-delivery") ?? "",
    name: request.headers.get("x-github-event") as never,
    signature,
    payload: body,
  });
  return Response.json({ ok: true });
}
```

## Step 6: Dashboard UI

Build a Next.js dashboard with three views:
- **Test Suite Editor**: Textarea for plain English descriptions, button to generate Playwright code, side-by-side preview of generated code
- **Run History**: Table of test runs per repo/branch, pass/fail/healed counts, click to view screenshots
- **Healing Log**: List of auto-healed tests with diffs (original vs healed code), confidence scores, ability to reject a heal

Use shadcn Table, Badge, Dialog, and Tabs components. Color code: green for passed, red for failed, amber for healed.

## Step 7: Deploy

```bash
# Frontend + API on Vercel
vercel deploy

# Required environment variables:
# ANTHROPIC_API_KEY=sk-ant-...
# GITHUB_APP_ID=12345
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA..."
# GITHUB_WEBHOOK_SECRET=your-secret
# NEXT_PUBLIC_SUPABASE_URL=
# SUPABASE_SERVICE_ROLE_KEY=

# For production test execution, use Docker + AWS ECS Fargate:
# docker build -t stably-runner .
# Playwright browsers pre-installed in the image (~300MB each)
# Store screenshots/videos in S3
```

GitHub Actions snippet:
```yaml
name: Stably Tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: stablyai/stably-runner-action@v1
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          suite-id: ${{ vars.STABLY_SUITE_ID }}
```

## Key Insights
- The `.describe()` annotation on Playwright locators is the architectural cornerstone — bake intent into every selector at generation time so the healer has context
- Separate healing into two tiers: action-level (fix selector during run) and maintenance-level (open PR to update source code)
- Use a cheaper model for test generation (Haiku), smarter model for healing (Sonnet) — cost matches complexity
- Diff-aware test execution (only run tests affected by the PR diff) is what keeps CI fast enough that developers actually use it

## Gotchas
- Playwright browsers are ~300MB each. Pre-install in your Docker image — never install at runtime in CI
- Never store full DOM snapshots in your database. Extract only interactive elements before sending to the LLM
- Visual snapshot healing is hard to get right. Start with selector healing only, add visual assertions once your core loop is stable
- GitHub App webhook delivery can be up to 30s delayed. Use the GitHub Actions runner approach as fallback