The cheapest pipeline is the one you delete. Databricks says Eclipse Aerospace scrapped its Lambda to Avro hop and now decodes proprietary flight telemetry directly in Spark, covering nearly 20 years, more than 3TB and over one million flight hours.
The Databricks blog frames it as a connector story, and that framing holds. This is about reading data where you query it.
Lambda was the tax.
Before PySpark custom data sources in Databricks Runtime 15.4 LTS, Eclipse Aerospace had three bad choices, per the account. Python UDFs risked out of memory on variable length messages. A Scala decoder required dedicated, always on compute. The chosen workaround decoded to Avro on AWS Lambda, which added latency, stored both binary and Avro copies, and hid failures in Lambda logs outside Spark UI metrics.
The new path keeps the proven Rust decoder and adds a thin Python skin. The Rust core takes bytes and returns structured records, with no knowledge of Spark, while a PyO3 and Maturin glue layer builds Arrow RecordBatches with arrow-rs and passes them through the Arrow C Data Interface. No bytes are re-encoded and no per row Python objects are built, so buffers Rust allocates are the buffers Python reads, and Spark consumes batches directly to Delta tables.
Performance is the reason to copy this pattern. The Rust backed source ran twice as fast as a similar PyArrow based pure Python source and nearly 20 times as fast as native Python without Arrow, according to benchmarks cited in the post. The API behind it shipped in Apache Spark 4.0 and is generally available from Runtime 15.4 LTS and Serverless environment 2, with an ARINC 429 example repo for testing.
I read this as validation for Python as control plane and Rust as decode engine. The fair concession is packaging, since a compiled wheel demands care around version and architecture and rewards proper CI and CD, as the example repo shows. That cost survives scrutiny because it replaces duplicate schemas, duplicate storage and two places to debug.
Do not confuse the jet maker with the rocket of the same name. Northrop Grumman says its Eclipse vehicle retains Antares flight proven systems and uses a 5.4 meter class fairing, a different reuse play in a different Eclipse. Serverless also has edges, since Databricks publishes numerical and API rate limits in its resource limits documentation, so on demand does not mean unlimited.
If your moat lives in a weird file format, stop copying it and teach Spark to read it.
