#Apache Spark
14 articles with this tag
Snowflake Simplifies Spark Migration
Snowflake's new CoCo tool simplifies the migration of Apache Spark workloads, aiming to ease data pipeline transitions for businesses.
Dow's Carbon Ledger Accelerates Sustainability
Dow leverages Databricks to create a Carbon Footprint Ledger, drastically cutting PCF calculation time and boosting sustainability transparency.
Pinterest's Medic AI Tames Spark Failures
Pinterest's Drasko Profirovic details Medic, an AI agent designed to diagnose and fix Apache Spark job failures, highlighting the evolution from prototype to a sophisticated multi-agent system.
Spark 4.2: AI-Native Analytics Arrive
Apache Spark 4.2 integrates AI-native analytics, enhances remote access via Spark Connect, and streamlines data processing with Auto CDC.
Databricks Unifies Lakehouse with Managed Tables
Databricks enhances its Unity Catalog managed tables, enabling external engines for read/write access and boosting performance via Predictive Optimization.
Databricks Spatial SQL Goes Live
Databricks Spatial SQL is now generally available, bringing native geospatial data processing, improved performance, and integrated map visualizations to its Lakehouse platform.
Spark's Real-Time Mode Powers Gaming
Databricks' Apache Spark Real-Time Mode with transformWithState now enables sub-second latency for gaming session tracking, eliminating complex architectures.
LinkedIn Sales Navigator Search Speed Boost
LinkedIn engineers drastically cut Sales Navigator's search data processing time by optimizing its Spark pipeline, enabling faster results for users.
Databricks Powers Global Health Volunteer Matching
Databricks for Good and the Virtue Foundation are using AI to map global healthcare resources and connect medical volunteers to critical needs.
Spark Streams Tackle Fraud in Milliseconds
Databricks' Spark Real-Time Mode and Lakebase offer a unified platform for sub-second fraud detection, eliminating complex infrastructure.
Databricks Reimagines Serverless Compute
Databricks is overhauling distributed systems for serverless compute, enhancing stability and performance through Spark Connect, intelligent routing, and adaptive autoscaling.
Spark Streaming Hits Millisecond Latency
Databricks' Apache Spark Structured Streaming real-time mode is now GA, offering sub-second latency and consolidating streaming needs onto a single engine.
Spark Drops Microbatch for Real-Time
Apache Spark's Real-Time Mode (RTM) breaks microbatch barriers, enabling millisecond latency for streaming workloads with a new hybrid execution model.
Spark Ditches Dual Engines for Real-Time Mode
Databricks' new Real-Time Mode for Spark aims to deliver sub-second streaming speeds, eliminating the need for separate processing engines.