# Rust plus Arrow beats the Lambda detour

_Eclipse Aerospace ditched Lambda-to-Avro and decodes flight data in Spark with Rust and Arrow._

**Published:** 2026-11-12
**Source:** https://www.startuphub.ai/ai-news/technology/2026/rust-plus-arrow-beats-the-lambda-detour

---

The cheapest pipeline is the one you delete. [Databricks](/startups/databricks) says Eclipse Aerospace scrapped its Lambda to Avro hop and now decodes proprietary flight telemetry directly in Spark, covering nearly 20 years, more than 3TB and over one million flight hours.

The [Databricks blog](https://www.databricks.com/blog/eclipse-aerospace-simplifies-flight-data-ingestion-databricks) frames it as a connector story, and that framing holds. This is about reading data where you query it.

Lambda was the tax.

Before PySpark custom data sources in [Databricks](/startups/databricks) Runtime 15.4 LTS, Eclipse Aerospace had three bad choices, per the account. Python UDFs risked out of memory on variable length messages. A Scala decoder required dedicated, always on compute. The chosen workaround decoded to Avro on AWS Lambda, which added latency, stored both binary and Avro copies, and hid failures in Lambda logs outside Spark UI metrics.

The new path keeps the proven Rust decoder and adds a thin Python skin. The Rust core takes bytes and returns structured records, with no knowledge of Spark, while a PyO3 and Maturin glue layer builds Arrow RecordBatches with arrow-rs and passes them through the Arrow C Data Interface. No bytes are re-encoded and no per row Python objects are built, so buffers Rust allocates are the buffers Python reads, and Spark consumes batches directly to Delta tables.

Performance is the reason to copy this pattern. The Rust backed source ran twice as fast as a similar PyArrow based pure Python source and nearly 20 times as fast as native Python without Arrow, according to benchmarks cited in the post. The API behind it shipped in Apache Spark 4.0 and is generally available from Runtime 15.4 LTS and Serverless environment 2, with an ARINC 429 example repo for testing.

I read this as validation for Python as control plane and Rust as decode engine. The fair concession is packaging, since a compiled wheel demands care around version and architecture and rewards proper CI and CD, as the example repo shows. That cost survives scrutiny because it replaces duplicate schemas, duplicate storage and two places to debug.

Do not confuse the jet maker with the rocket of the same name. [Northrop Grumman says](https://www.northropgrumman.com/what-we-do/space/eclipse) its Eclipse vehicle retains Antares flight proven systems and uses a 5.4 meter class fairing, a different reuse play in a different Eclipse. Serverless also has edges, since [Databricks](https://www.startuphub.ai/startups/databricks) publishes numerical and API rate limits in its [resource limits documentation](https://docs.databricks.com/aws/en/resources/limits), so on demand does not mean unlimited.

If your moat lives in a weird file format, stop copying it and teach Spark to read it.

---

Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.

© StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.