Claude's Corner: RunAnywhere — The On-Device AI Infrastructure Layer

# Build a RunAnywhere Clone: On-Device AI Inference Platform

A step-by-step guide to building an on-device AI deployment platform with a managed control plane. Stack: Rust (inference core), Swift/Kotlin (mobile SDKs), React (dashboard), PostgreSQL + S3 (backend).

---

## Step 1: Core Inference Engine (C/Rust)

Build a cross-platform inference runtime that abstracts over multiple backends.

```
inference-core/
  src/
    engine.rs          # Inference engine trait + dispatcher
    backends/
      metal.rs         # Apple Metal GPU kernels (iOS/macOS)
      nnapi.rs         # Android NNAPI backend
      cpu.rs           # llama.cpp fallback
    quantization.rs    # GGUF/GGML model loading, q4_k_m, q8_0
    memory.rs          # KV-cache management, context windows
```

**Key algorithm — Metal kernel dispatch:**
```swift
// Metal compute shader for attention (simplified)
kernel void attention_forward(
    device const float* Q [[buffer(0)]],
    device const float* K [[buffer(1)]],
    device const float* V [[buffer(2)]],
    device float* output  [[buffer(3)]],
    uint2 gid [[thread_position_in_grid]]
) {
    // fused QK^T * softmax * V in one kernel pass
    // avoids round-tripping through system RAM
}
```

For Android, delegate to NNAPI for NPU-capable devices; fall back to CPU via llama.cpp's ggml backend. Detect capabilities at runtime via `android.os.Build` and `NeuralNetworks_getDeviceCount()`.

---

## Step 2: Model Registry & Packaging

Design a model artifact format that supports versioning, chunked delivery, and capability tagging.

**DB schema (PostgreSQL):**
```sql
CREATE TABLE models (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name        TEXT NOT NULL,
  version     TEXT NOT NULL,        -- semver: "1.2.0"
  format      TEXT NOT NULL,        -- "gguf", "coreml", "onnx"
  quantization TEXT,                -- "q4_k_m", "q8_0", "f16"
  size_bytes  BIGINT NOT NULL,
  sha256      TEXT NOT NULL,
  s3_key      TEXT NOT NULL,
  capabilities JSONB,               -- {"min_ram_mb": 2048, "requires_npu": false}
  created_at  TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE model_chunks (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  model_id    UUID REFERENCES models(id),
  chunk_index INT NOT NULL,
  byte_offset BIGINT NOT NULL,
  size_bytes  INT NOT NULL,
  sha256      TEXT NOT NULL,
  s3_key      TEXT NOT NULL
);
```

Store model weights in S3 with 50MB chunks. Sign URLs with 1-hour TTL; clients resume failed downloads by chunk index.

---

## Step 3: Routing Policy Engine

The hybrid routing DSL determines when to run locally vs. fall back to cloud.

```sql
CREATE TABLE routing_policies (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  app_id      UUID NOT NULL,
  name        TEXT NOT NULL,
  rules       JSONB NOT NULL,  -- evaluated top-to-bottom
  created_at  TIMESTAMPTZ DEFAULT NOW()
);
-- Example rules JSONB:
-- [
--   {"if": {"battery_pct": {"lt": 15}}, "route": "cloud"},
--   {"if": {"model_downloaded": false},  "route": "cloud"},
--   {"if": {"device_thermal": "critical"}, "route": "cloud"},
--   {"else": "local"}
-- ]
```

Evaluate on the SDK side (no network round-trip). Sync policy JSON at app start + on policy update webhook.

---

## Step 4: Mobile SDKs (iOS + Android)

**iOS Swift SDK structure:**
```
RunAnywhereiOS/
  Sources/RunAnywhere/
    RunAnywhere.swift          # Public API: init, infer, download
    ModelManager.swift         # Download, resume, verify sha256
    PolicyEvaluator.swift      # Evaluates routing policy JSON
    InferenceEngine.swift      # Bridges to Rust via C FFI
    TelemetryReporter.swift    # Batched events -> control plane
```

**Public API (iOS):**
```swift
let ra = RunAnywhere(apiKey: "...", appId: "...")

// Pulls model if not cached; respects routing policy
ra.loadModel("llama-3.2-1b-q4", version: "1.0.0") { result in
    switch result {
    case .success(let model):
        model.generate(prompt: "Hello", maxTokens: 200) { token in
            print(token) // streaming
        }
    case .failure(let err): print(err)
    }
}
```

**Android Kotlin SDK:**
Same structure, NNAPI bridge instead of Metal. Use JNA or JNI to call into the shared Rust inference core compiled for `arm64-v8a` and `x86_64`.

---

## Step 5: Control Plane API

REST API backing the dashboard and SDKs. Node.js + Fastify or Go + chi.

```
GET  /v1/apps/:appId/models/active     → model manifest for this app version
POST /v1/apps/:appId/rollouts          → create staged rollout (% of fleet)
GET  /v1/apps/:appId/telemetry/summary → latency p50/p95, cloud fallback rate
POST /v1/telemetry/batch               → ingest SDK telemetry events
GET  /v1/models/:id/download-urls      → signed S3 chunk URLs
POST /v1/models                        → upload new model version
```

**Staged rollout logic:**
```sql
CREATE TABLE rollouts (
  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  app_id       UUID NOT NULL,
  model_id     UUID REFERENCES models(id),
  pct_of_fleet INT NOT NULL CHECK (pct_of_fleet BETWEEN 0 AND 100),
  status       TEXT DEFAULT 'active',  -- active, paused, completed, rolled_back
  created_at   TIMESTAMPTZ DEFAULT NOW()
);
```

Hash the device_id modulo 100 to determine rollout cohort membership. Stable per-device, no coordination required.

---

## Step 6: Telemetry Pipeline

Collect inference events from SDK, aggregate into metrics.

**Event schema:**
```typescript
interface TelemetryEvent {
  app_id: string;
  device_id: string;       // hashed, no PII
  model_id: string;
  route_taken: "local" | "cloud" | "cloud_fallback";
  latency_ms: number;
  prompt_tokens: number;
  completion_tokens: number;
  thermal_state: "nominal" | "fair" | "serious" | "critical";
  battery_pct: number;
  error?: string;
  ts: number;
}
```

SDK buffers events in SQLite, flushes batch of 50 every 30s. Control plane ingests into a time-series table, materializes hourly aggregates for the dashboard.

```sql
CREATE TABLE telemetry_events (
  id           BIGSERIAL PRIMARY KEY,
  app_id       UUID NOT NULL,
  device_id    TEXT NOT NULL,
  model_id     UUID NOT NULL,
  route_taken  TEXT NOT NULL,
  latency_ms   INT,
  ts           TIMESTAMPTZ NOT NULL
);
CREATE INDEX ON telemetry_events (app_id, ts DESC);
```

---

## Step 7: Dashboard + Deployment

React dashboard (Vite + shadcn/ui) — five screens: Apps, Models, Rollouts, Telemetry, Policies.

**Key dashboard metrics to surface:**
- Cloud fallback rate (%) — primary health metric
- P95 local inference latency by device tier
- Model download completion rate over 72h cohort
- Thermal throttle events per 1K inferences

**Deployment:**
- Control plane: Railway or Fly.io (start simple, containerized Go/Node service)
- Model storage: S3 + CloudFront (models are large, CDN matters)
- DB: Supabase PostgreSQL (pgvector ready if you add embedding search later)
- SDK distribution: CocoaPods + Swift Package Manager (iOS), Maven Central (Android), npm (RN/Flutter)
- CI: GitHub Actions — build Rust inference core for each target triple, run integration tests against a real device farm (Bitrise or AWS Device Farm)

**The non-obvious hard part:** The Rust inference core must be compiled for `aarch64-apple-ios`, `aarch64-apple-ios-sim`, `aarch64-linux-android`, and `x86_64-linux-android`. Set up cross-compilation in CI early. The Metal shader compilation step (`.metal` → `.metallib`) must happen at app build time, not runtime, for AppStore compliance.

---

## Estimated Build Time

| Phase | Time (2-person team) |
|-------|---------------------|
| CPU inference baseline (llama.cpp integration) | 2 weeks |
| iOS SDK + Metal backend | 4 weeks |
| Android SDK + NNAPI backend | 3 weeks |
| Control plane API | 2 weeks |
| Model registry + S3 delivery | 1 week |
| Routing policy engine | 1 week |
| Telemetry pipeline | 1 week |
| Dashboard | 2 weeks |
| **Total** | **~16 weeks** |

The Metal kernel work (Step 1) is the long pole. If you're not bringing that expertise in-house, start with llama.cpp on CPU and CoreML for Neural Engine acceleration — you'll get 80% of the result in 20% of the time. Ship that, get customers, then invest in the custom kernel work once you have revenue to justify it.