Claude's Corner: Fern - From Sales AI to Robot World Models

Fern (formerly Ishiki Labs) is a YC W2026 startup that pivoted from socially-aware meeting AI to custom robot world models for RL policy training. Here is how they did it, and how you can build a clone.

Daniel Singer

Editor, StartupHub.ai

Aug 18 at 11:10 AM8 min read

Fern homepage screenshot with Claude's Corner badge

TL;DR

Fern (formerly Ishiki Labs) is a YC W2026 startup that pivoted from socially-aware meeting AI to building custom robot world models for reinforcement learning policy training. Founded by two ex-Meta AI researchers, they target the simulation bottleneck slowing physical AI deployment. The moat is research-grade world model construction baked into a product.

6.6

Build difficulty

Contents(7)

TL;DR

Fern (formerly Ishiki Labs) is a YC W2026 startup that pivoted from socially-aware meeting AI to building custom robot world models for reinforcement learning policy training. Founded by two ex-Meta AI researchers with deep multimodal and low-latency infrastructure experience, they are targeting the simulation bottleneck slowing down every robotics company deploying physical AI. The moat is research-grade world model construction baked into a product, not just another physics engine wrapper.

The Pivot Most People Missed

When Ishiki Labs launched at YC W2026 Demo Day, the pitch was a socially-aware AI meeting assistant. Real-time coaching during sales calls. An invisible copilot that knew when to stay silent. It was a polished story - two ex-Meta researchers, a PhD from Purdue, low-latency multimodal systems experience from building smart glasses. The technical chops were obvious.

And then they scrapped it.

Not completely - the core insight survived. Amit Yadav spent years at Meta working on AI that had to know when not to talk. Multimodal assistants on smart glasses cannot interrupt every conversation; they have to model social context in real time. Robert Xu spent four years building the infrastructure to run those systems at sub-100ms latency on hardware smaller than a phone. That is a very specific skill set.

Turns out it maps almost directly onto a harder, less crowded problem: building world models for robots.

Today, fern.bot leads with a single line - "Enabling physical AI at scale" - and a product that helps robotics companies evaluate and train robot policies in simulation without burning through physical hardware cycles. The meeting AI is gone. The technical DNA is the same. This is the startup worth writing about.

What They Build

Robotics companies face a brutal iteration cycle. You train a policy in simulation, test it on real hardware, watch it fail in ways the simulator did not predict, debug for weeks, repeat. The sim-to-real gap - the difference between how a robot behaves in a physics simulator versus the physical world - has been one of the core unsolved problems in applied robotics for a decade.

The standard tools are physics engines: MuJoCo, Isaac Sim, PyBullet. They are accurate for contact dynamics but generic. They do not know what your specific robot looks like, how your specific end effector deforms on impact, or what the acoustic properties of your warehouse floor do to vibration-sensitive joint readings. Every robotics team ends up spending engineering time tuning simulation parameters to match their real-world setup, and it is never quite right.

Fern's approach is different: instead of a generic physics simulator, they build a custom world model trained on your specific robot and environment. High-fidelity, learned from data, tuned to close the sim-to-real gap for your hardware specifically. The pitch to robotics companies: test and train your policies entirely in software before you burn real robot time.

The product is B2B, delivered as a platform with a whitepaper and case study trail - not a consumer product or API-first tool. The sales motion involves meeting the team, which tracks for selling to robotics engineering leads who need to trust the simulation environment before betting RL training time on it.

How It Works

The architecture has three layers, and each draws directly on the founders' Meta experience.

Foundation model layer. Fern trains a base world model on diverse robot interaction data - sensor readings, actuator commands, outcome sequences across multiple robot form factors and environments. This is the research-heavy part. Amit's background training multimodal models at Meta's LLaMA team is load-bearing here. Building a foundation model that generalizes across robot types requires the same attention to training data diversity and evaluation methodology as any large-scale ML research project.

Custom fine-tuning layer. Once the foundation exists, you fine-tune on your specific robot's interaction data. This is where "high-fidelity, custom" comes in. The fine-tuned model captures the quirks of your hardware - the exact slippage on your gripper, the latency profile of your joint controllers, the visual appearance of your specific working environment. The output is a world model calibrated to your robot in a way no off-the-shelf simulator can match.

Inference and evaluation layer. This is where Robert Xu's work at Meta Reality Labs and Citadel Securities applies directly. Running a world model fast enough for RL training requires the same kind of low-latency, high-throughput infrastructure he spent four years building for smart glasses. The simulation has to run faster than real time, cheaply enough on GPU that the economics work. That is a hard infrastructure problem, not just a research problem.

The product competes indirectly with physics simulators and directly with teams building in-house simulation environments. Neither is sitting still - NVIDIA's Omniverse ecosystem is large, and several well-funded startups are working on similar world model approaches. Fern's differentiator is the custom fine-tuning: they are not trying to build a general-purpose simulator, they are trying to be better than your internal team for your specific robot.

Difficulty Score

Building this stack is not a weekend project. Here is how the layers break down:

ML/AI (9/10): Training a foundational world model that generalizes across robot types and then fine-tunes to specific hardware is frontier research territory. The research methodology, evaluation setup, and model architecture decisions are the entire product.
Data (8/10): Getting high-quality robot interaction data at scale, with the diversity needed to train a foundation model, is genuinely hard. It requires either partnerships with robotics companies (who guard operational data closely) or building data collection infrastructure from scratch.
Backend (7/10): Running world models fast enough for RL training loops requires serious GPU infrastructure and inference optimization. Robert Xu's Citadel background building latency-sensitive systems is directly applicable here.
Frontend (3/10): B2B platform portal, not a consumer product. Functional over flashy.
DevOps (6/10): GPU cluster management, simulation pipeline orchestration, data pipelines for fine-tuning. Standard MLOps complexity at research scale.

Average difficulty: 6.6 out of 10. The ML and data layers make this genuinely hard to replicate. The infrastructure is solvable with the right team and budget.

The Moat

The real moat is not the world model itself - foundation model training techniques are well-documented and several labs are working on robot world models. The moat is the fine-tuning data flywheel.

Every robotics customer Fern onboards generates data that improves the foundation model. More customers means more robot types, more environments, more edge cases covered. A company that has fine-tuned world models for 50 different robot platforms has a calibration dataset a new entrant cannot buy or replicate quickly. This is the same compounding dynamic that made large language model providers hard to compete with once they hit critical data mass.

What is easy to copy: the general idea of training a neural world model instead of using a physics engine. Several academic groups and at least a handful of well-funded startups are working on this. The concept is not proprietary.

What is hard to copy: the custom fine-tuning methodology, the data collection relationships with robotics customers, and - frankly - the team. Amit's combination of multimodal model research experience and product instincts is rare. Finding another pair of researchers with Meta Reality Labs infrastructure depth and a PhD-level ML research track record, who have already been through a pivot and come out with a sharper thesis, takes time.

StartupHub.ai data shows Fern scores 54 out of 100 on our composite startup signal - well above the 35.8 average across the 1,305 robotics startups we track. For reference, Physical Intelligence scores 65 and Skild AI sits at 64. Fern is early-stage but trending in the right direction for a two-person team that has already pivoted once.

Replicability Score: 72 / 100

Hard, but not impossible. The research methodology for training robot world models is increasingly well-understood - you could hire a strong ML team and build a version of this. The real barrier is the data and the calibration expertise. Getting enough diverse robot interaction data to train a useful foundation model requires either deep industry partnerships or a very patient capital strategy. The founders' Meta background accelerates credibility with potential enterprise customers in a way that is hard to fake.

The pivot from meeting AI to physical AI also reveals something about how Amit and Robert think: they are hunting for the hardest version of the problem they are technically equipped to solve, not the most defensible business position at a given moment. That is the kind of founder behavior that makes startups hard to predict and even harder to compete with.

If you are building in this space, the order of operations matters: start with a specific robot platform and environment, prove the sim-to-real gap closes, then expand to foundation model territory. Do not try to boil the ocean on day one.

© 2026 StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training requires a license. See our terms.

Build This Startup with Claude Code

Complete replication guide, install as a slash command or rules file

# Building a Robot World Model Platform: Step-by-Step with Claude Code

## Step 1: Define the Data Schema

Create a PostgreSQL schema to track robots, simulation sessions, world model versions, and policy evaluation results.

```sql
CREATE TABLE robots (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name text NOT NULL,
  form_factor text,
  sensor_config jsonb,
  created_at timestamptz DEFAULT now()
);

CREATE TABLE interaction_episodes (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  robot_id uuid REFERENCES robots(id),
  episode_data jsonb NOT NULL,
  environment_id text,
  duration_ms integer,
  recorded_at timestamptz DEFAULT now()
);

CREATE TABLE world_model_versions (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  robot_id uuid REFERENCES robots(id),
  model_uri text NOT NULL,
  training_config jsonb,
  eval_metrics jsonb,
  created_at timestamptz DEFAULT now()
);

CREATE TABLE policy_evaluations (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  policy_uri text NOT NULL,
  world_model_id uuid REFERENCES world_model_versions(id),
  task_spec jsonb,
  success_rate float,
  mean_return float,
  created_at timestamptz DEFAULT now()
);
```

## Step 2: Build the Data Ingestion Pipeline

Set up an API to ingest robot interaction episodes from physical hardware.

```python
from fastapi import FastAPI, UploadFile
import boto3

app = FastAPI()
s3 = boto3.client("s3")

@app.post("/episodes/ingest")
async def ingest_episode(robot_id: str, file: UploadFile):
    raw = await file.read()
    episode = parse_episode(raw)
    key = f"episodes/{robot_id}/{episode.id}.pkl"
    s3.put_object(Bucket="robot-data", Key=key, Body=raw)
    await db.execute(
        "INSERT INTO interaction_episodes (robot_id, episode_data) VALUES ($1, $2)",
        robot_id, episode.to_jsonb()
    )
    return {"episode_id": episode.id}
```

## Step 3: Train the Foundation World Model

Use a transformer-based architecture to predict next state given current state + action.

```python
import torch
from torch import nn

class WorldModel(nn.Module):
    def __init__(self, obs_dim: int, action_dim: int, latent_dim: int = 512):
        super().__init__()
        self.encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=latent_dim, nhead=8), num_layers=6
        )
        self.dynamics = nn.GRU(latent_dim + action_dim, latent_dim, batch_first=True)
        self.decoder = nn.Linear(latent_dim, obs_dim)

    def forward(self, obs_seq, action_seq):
        z = self.encoder(obs_seq)
        z_next, _ = self.dynamics(torch.cat([z, action_seq], dim=-1))
        return self.decoder(z_next)
```

## Step 4: Build the Fine-Tuning Service

Expose a fine-tuning endpoint that customers hit with their specific robot data.

```python
from peft import LoraConfig, get_peft_model

def fine_tune_for_robot(robot_id: str, base_model_path: str, customer_episodes: list):
    base_model = WorldModel.load(base_model_path)
    config = LoraConfig(r=16, lora_alpha=32, target_modules=["dynamics"])
    model = get_peft_model(base_model, config)
    customer_loader = DataLoader(customer_episodes, batch_size=32)
    fine_tune(model, customer_loader, epochs=20)
    model_uri = f"s3://models/{robot_id}/adapter_latest.bin"
    model.save_pretrained(model_uri)
    return model_uri
```

## Step 5: Build the Gym-Compatible RL Environment

Wrap the world model so robotics teams can plug into any RL framework.

```python
import gymnasium as gym
import numpy as np

class WorldModelEnv(gym.Env):
    def __init__(self, world_model, robot_id: str):
        self.model = world_model
        self.state = None
        self.observation_space = gym.spaces.Box(low=-np.inf, high=np.inf, shape=(512,))
        self.action_space = gym.spaces.Box(low=-1, high=1, shape=(32,))

    def reset(self, seed=None):
        self.state = self.model.sample_initial_state()
        return self.state, {}

    def step(self, action):
        next_state = self.model.predict_next_state(self.state, action)
        reward = self.compute_reward(next_state)
        done = self.is_terminal(next_state)
        self.state = next_state
        return next_state, reward, done, False, {}
```

## Step 6: Deploy the Inference Stack

Serve the world model for real-time RL training loops using TensorRT for 3-5x latency reduction.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: world-model-inference
spec:
  replicas: 4
  template:
    spec:
      containers:
      - name: inference-server
        image: fern-inference:latest
        resources:
          limits:
            nvidia.com/gpu: 1
        env:
        - name: INFERENCE_BACKEND
          value: "tensorrt"
        - name: MAX_BATCH_SIZE
          value: "256"
```

## Step 7: Build the Sim-to-Real Gap Dashboard

Track policy performance inside the world model vs. real hardware to prove and improve transfer.

```typescript
export function PolicyEvalChart({ evaluations }) {
  const simResults = evaluations.filter(e => e.environment === "sim");
  const realResults = evaluations.filter(e => e.environment === "real");

  return (
    <div className="eval-dashboard">
      <h2>Sim vs. Real Success Rate</h2>
      <LineChart data={[
        { label: "World Model Sim", data: simResults.map(e => e.success_rate) },
        { label: "Real Hardware", data: realResults.map(e => e.success_rate) }
      ]} />
      <p>Sim-to-real gap: {calculateGap(simResults, realResults).toFixed(1)}%</p>
    </div>
  );
}
```

Closing this feedback loop is how you prove (and continuously improve) sim-to-real transfer quality.

Install for:

claude-code-skills.md

#robotics #physical-ai #world-models #yc-w2026 #simulation #reinforcement-learning #deep-learning

Written by

Daniel Singer

Editor, StartupHub.ai

Daniel Singer is the editor of StartupHub.ai, a technology expert and thought leader on AI and its applications across sectors, from fintech and healthcare to developer tooling and consumer software. He writes and tests the tools covered here thoroughly and regularly, and built StartupHub.ai to give founders, operators and buyers a clearer read on what they are actually being sold.