#Data Engineering
50 articles with this tag
Databricks Unlocks Database Evolution
Databricks Lakebase's new database branching capabilities make evolutionary database development principles a reality at scale.
Spark's Real-Time Mode Powers Gaming
Databricks' Apache Spark Real-Time Mode with transformWithState now enables sub-second latency for gaming session tracking, eliminating complex architectures.
Snowflake Streams for Real-Time AI
Snowflake enhances its platform with Datastream for Kafka-compatible streaming, AI-powered tools, and expanded data integration capabilities to fuel agentic AI.
Snowflake's Adaptive Compute
Snowflake's new Adaptive Compute technology dynamically scales resources for data workloads, promising higher performance and reduced operational complexity.
Snowflake CoCo Goes Everywhere
Snowflake's AI coding agent, CoCo, is expanding beyond its data cloud with desktop, mobile, and Slack integrations, aiming to embed governed AI development everywhere.
Databricks Lakebase: Database Branching Reimagined
Databricks Lakebase's new database branching feature allows developers isolated, production-like environments, streamlining database evolution.
Databricks Shines at SIGMOD 2026
Databricks' Enzyme engine and Spark Declarative Pipelines are showcased at SIGMOD 2026, simplifying complex data engineering tasks.
Cloudflare's AI Data Agent
Cloudflare unveils Town Lake, a unified data platform, and Skipper, its AI agent for natural language data querying, enhancing internal data access and governance.
Databricks Unifies Operational Data
Databricks' new Lakebase Change Data Feed simplifies operational data integration into the Lakehouse, enabling direct streaming and unified governance.
Octopus Energy Slashes Costs 50x
Octopus Energy slashed data engineering costs by 50x to meet UK's MHHS regulation, processing 98.8% fewer data rows and improving freshness.
LinkedIn Sales Navigator Search Speed Boost
LinkedIn engineers drastically cut Sales Navigator's search data processing time by optimizing its Spark pipeline, enabling faster results for users.
Databricks Unifies Data, Analytics, and AI
Databricks aims to simplify data operations with its unified Lakehouse Platform, integrating data warehousing, analytics, and AI development.
Snowflake Turbocharges dbt with Fusion
Snowflake integrates dbt Fusion, boosting compilation speeds for complex projects with no added cost or reconfiguration required.
Databricks Launches Analytics Engineer Path
Databricks launches a new learning pathway for SQL practitioners to become analytics engineers, covering data modeling, pipelines, and AI agent deployment.
Snowflake Simplifies Data Pipelines
Snowflake introduces DCM Projects and Cortex Code for declarative data pipelines, simplifying workflow management and reducing manual coding.
Databricks Unifies Clinical Data
Databricks' new open-source Site Feasibility Workbench brings clinical trial intelligence onto its Lakehouse, tackling data silos and improving auditability.
DataMaster: Autonomous Data Engineering
DataMaster pioneers autonomous data engineering, unlocking significant ML gains by optimizing data pipelines rather than algorithms, as shown on MLE-Bench Lite and PostTrainBench.
Databricks Syncs Postgres to Lakehouse Natively
Databricks unveils Native Lakehouse Sync, directly replicating operational Postgres data into Unity Catalog without pipelines, simplifying AI and analytics integration.
Databricks Driver Gets Faster, Feature-Rich
Databricks rolls out its open-source JDBC driver with significant speed boosts and new features for enhanced data connectivity.
Databricks Taps Students for AI Leadership
Databricks launches Student Fellows program to equip university students with data and AI skills, fostering campus leadership and career opportunities.
Databricks boosts Postgres writes 5x
Databricks' lakebase architecture boosts Postgres write performance up to 5x by offloading durability tasks to distributed storage.
Databricks Reimagines Serverless Compute
Databricks is overhauling distributed systems for serverless compute, enhancing stability and performance through Spark Connect, intelligent routing, and adaptive autoscaling.
Snowflake's AI Boost for Data Integration
Snowflake fuses its Openflow data integration service with its Cortex Code AI agent to streamline AI-driven data pipelines.
Databricks Crushes Monitoring Scale
Databricks scales monitoring to 10 trillion samples/day using customized open-source tools and a new Lakehouse platform for cost-effective high-cardinality data.
Backstage Ditches Postgres for Databricks Lakebase
Databricks integrates Spotify's Backstage with Lakebase, enabling instant database branching and transforming developer workflows by eliminating operational vs. analytical database silos.
Databricks Speeds Up Analytics with Sketch Functions
Databricks enhances analytics with new sketch functions, delivering orders-of-magnitude speedups for percentile, distinct count, and top-K queries.
Healthcare Data: From Months to Minutes
Databricks and Redox cut clinical data integration times from months to minutes with natural language prompts and subsecond data streaming.
Snowflake Kafka Connector V4 Arrives
Snowflake's Kafka Connector V4 is here, moving ingestion logic server-side for major performance and cost gains.
Databricks Touts AI-Powered Data Pipelines
Databricks unveils Genie Code, an AI agent for Lakeflow, to automate data pipeline creation, orchestration, and debugging via natural language.
AI Needs Faster Databases
AI demands real-time data. Traditional operational databases lag, but new 'lakebase' architectures are bridging the gap for faster, smarter AI.
LLM Agents Tackle Database Joins
Databricks tests LLM agents for SQL join order optimization, achieving significant performance gains over traditional methods.
Snowflake Adds Declarative Infra Management
Snowflake's new DCM Projects feature allows users to manage data infrastructure declaratively, bringing software engineering practices to Snowflake object management.
Snowflake's AI Coder Goes Full Stack
Snowflake's Cortex Code AI assistant now understands and integrates across a broader data stack, offering specialized skills and embeddable platform capabilities.
Databricks Summit: AI Agents, Vibe Coding Take Center Stage
Databricks' Data + AI Summit 2026 in San Francisco will highlight AI agents and 'vibe coding', with new training and a Context Engineering certification.
Mercedes-Benz Cuts Data Costs 66% with Databricks
Mercedes-Benz slashed data egress costs by 66% using Databricks Delta Sharing and intelligent replication, building a secure cross-cloud data mesh.
Snowflake's Cortex Code Tames dbt
Snowflake's Cortex Code streamlines dbt project development by automating model creation, testing, and optimization with an AI agent.
dbt on Databricks: Open Platform Advantage
Databricks is enhancing dbt workflows with its unified lakehouse, offering open foundations, integrated governance, and optimized performance to accelerate data transformation.
Databricks Unlocks Unstructured Data
Databricks enhances its platform with Document Intelligence and Lakeflow, enabling businesses to unlock and process vast amounts of unstructured enterprise data.
Snowflake Simplifies Iceberg Storage
Snowflake's new managed storage for Apache Iceberg tables offers the open format's interoperability with Snowflake's resilient, zero-management infrastructure.
Databricks Powers Real-Time Search
Databricks unveils its platform for building real-time product search, integrating Vector Search, Lakeflow, and Lakebase for ingestion, retrieval, and operational data.
Lovable Taps Databricks for Data Apps
Lovable integrates with Databricks, allowing non-technical users to build data-driven applications using natural language, eliminating backend bottlenecks.
Databricks Embraces Iceberg v3
Databricks previews Apache Iceberg v3 support, integrating Row Lineage, Deletion Vectors, and VARIANT for enhanced performance and interoperability in the open lakehouse.
Databricks Postgres branches like Git
Databricks Lakebase brings Git-style branching to Postgres, offering near-instant, isolated database environments via copy-on-write technology.
Databricks Unifies Data Workflows
Databricks introduces a unified platform to eliminate data silos and enhance collaboration across financial institutions using AI and governed analytics.
Snowflake Taps AI for Retail Scale
Snowflake Intelligence is empowering retailers like the Mark Anthony Group to scale AI, democratize data access, and drive business outcomes through generative BI.
Snowflake's AI Coder Goes Broad
Snowflake's Cortex Code AI agent is now widely available, featuring enhanced usability and new capabilities for data engineering.
Databricks A/B Testing Framework Powers Game Analytics
HARDlight leverages Databricks to automate A/B testing analysis, doubling experimentation capacity and building trust through standardized, LLM-enhanced insights.
Snowflake's AI Coding Agent Streamlines Data Engineering
Snowflake's new AI coding agent, Cortex Code, aims to simplify data pipeline creation and accelerate development for engineers and analysts.
Snowflake Boosts AI Data Sharing
Snowflake enhances its AI data sharing platform with new features for reliability, usability, and transparency, crucial for production AI.
Databricks AutoCDC Ends Hand-Coding Pain
Databricks AutoCDC automates change data capture and SCD pipelines, slashing manual coding, improving performance, and cutting costs with declarative simplicity.