Databricks is giving users more granular control over their data encryption with the introduction of customer-managed keys (CMK) for its Lakehouse Postgres offering. This move addresses a critical need for enhanced data security for regulated environments, allowing organizations to leverage their own Key Management Service (KMS) for protecting sensitive information.
Traditionally, cloud database encryption relies on provider-managed keys. Databricks' approach, detailed on their blog, shifts this responsibility, enabling users to employ keys from AWS KMS, Azure Key Vault, or Google Cloud KMS. This ensures the root of trust remains firmly within the customer's control, a crucial aspect for compliance.
Encryption Across the Lakehouse Stack
The architecture of Lakehouse Postgres separates storage and compute. This separation, while enabling scalability, presents a unique encryption challenge: both long-term storage and transient compute caches require robust protection.
Databricks employs a hierarchical envelope encryption model. Here, customer-managed keys (CMKs) reside in the user's cloud KMS and are never exposed to Databricks. Instead, Databricks receives encrypted versions of keys necessary for data decryption.
This hierarchy involves three tiers: the customer-managed root key (CMK), a Key Encryption Key (KEK) used by Databricks' Key Manager Service, and Data Encryption Keys (DEKs) unique to each data segment. When data access is required, components unwrap DEKs using keys obtained from the customer's KMS.
The daily brief
Get the free daily brief with the day's AI news, new AI startups and funding rounds every morning.
Join 64,000 other subscribers.
Revocation as a 'Kill Switch'
A significant implication of this model is the ability to revoke access. Upon CMK revocation, the unwrapping process fails, rendering data cryptographically inaccessible. This also triggers the termination of active compute instances, effectively destroying ephemeral keys and shredding local disk data.
This capability provides a powerful failsafe, particularly for high-compliance Postgres workloads. It transforms the KMS into a technical 'kill switch' for sensitive data.
Practical Implementation and Workflow
CMK implementation integrates with Databricks' existing account-to-workspace delegation model, separating duties between security administrators and data access personnel.
The process involves configuring the key at the account level, binding it to a specific workspace, and then applying it to all new Lakehouse projects within that workspace. This allows for granular control, with different workspaces potentially using distinct CMKs to meet multi-tenant or departmental security requirements.
Seamless key rotation is supported, allowing users to rotate their CMK in their cloud provider's console without requiring re-encryption of data or downtime. The cryptographic operations against the CMK are logged in the customer's cloud audit services, enhancing security auditability.
This advanced level of cryptographic control is now available for Databricks Enterprise tier customers, underscoring the platform's commitment to robust data sovereignty and security for demanding workloads.
