Kubernetes Security Goes Deep

Securing infrastructure at LinkedIn’s scale demands a comprehensive strategy, starting with trusted hardware and extending to every line of code. The company has detailed its approach to establishing a verifiable chain of trust for its Kubernetes workloads, a critical step in protecting user data and system integrity. This effort focuses on assigning every piece of software, from feed services to complex ML pipelines, a digital credential acting as a signed ID card.

This system aims to prevent 'identity spoofing' by attesting each workload's identity against an internal Identity Registry. The entire process is integrated into the software lifecycle, ensuring workloads are identified and secured from creation.

Building a Next-Generation Security Framework

As LinkedIn's infrastructure expands, so does the diversity of workloads requiring robust security. The challenge lies in securing varied systems like Flink streams, Airflow jobs, and third-party databases such as Couchbase and MySQL. To meet this, LinkedIn extended cert-manager, a popular open-source certificate management tool.

A key enabler is cert-manager's CSI driver, which injects certificates directly into workload containers as volumes. This method ensures private keys remain on the node, mitigating exfiltration risks.

The multi-cluster architecture presented a significant hurdle, as jobs orchestrated in one cluster can spawn workers in another. The workload identity system needed to operate consistently across these boundaries, demanding standardized certificate issuance and attestation.

cert-manager Integration at LinkedIn

LinkedIn utilizes cert-manager for automated certificate management, including issuance, rotation, and deletion. Crucially, it integrates seamlessly with an internal Identity Registry for strong workload attestation. The platform is adaptable, supporting custom issuers and approver policies.

Two modes cater to different workload needs: 'Fully Managed' for most deployments, activated by a simple label, and 'Self Serve' for manually deployed services or external platforms.

The 'Fully Managed' solution integrates open-source cert-manager components with internal systems: an Identity Registry for global unique identities, an HSM Service for secure key access, and a Mutating Admission Webhook to inject cert-manager's CSI volume. A custom component, Lipki-Controller, acts as a CertificateRequest Approver and Issuer, validating requests, performing attestation, and signing certificates.

Automated Workflow for Workload Identity

The automated workflow issues a unique digital identity upon workload creation. Deployment systems add a `spiffe: enabled` label, triggering the cert-manager mutating webhook.

This webhook injects a CSI volume, mounting it read-only into the application container with metadata like application name and issuer details.

As pods are scheduled, the cert-manager CSI driver creates a CertificateRequest (CR) referencing the workload and issuer. The Lipki-controller then validates the CR, performs workload attestation via the Identity Registry, and issues a signed certificate.

This signed certificate is securely and transparently mounted into the pod by the CSI driver.

Kyverno Policy Integration

Integrating with Kyverno policies adds crucial security guardrails. Two policies restrict certificate issuance to PKI team-managed resources and control who can create CertificateRequests via RBAC.

Adoption Strategy and Developer Experience

LinkedIn’s adoption strategy prioritized automated trust, zero-friction security, and a seamless transition. Every workload automatically receives attested identity certificates, enabling mutual TLS (mTLS) for encrypted, verified service interactions.

The goal was to eliminate developer toil by making secure configurations the default. This allows developers to focus on features while infrastructure handles security.

Full backward compatibility ensured platform stability during migration, allowing legacy and new certificates to coexist and providing rollback capabilities.

Real-time observability dashboards track adoption progress, identifying services needing migration. The rollout began with an opt-in model, using the `spiffe: enabled` label for SPIFFE-based certificates.

Reducing Developer Toil with Library Integrations

To simplify adoption, LinkedIn developed authentication libraries for Java, Go, and Rust. These libraries abstract the complexities of credential management, enabling secure mTLS connections.

Certificates and keys are mounted to restricted paths within containers, protected by file access controls. By integrating these libraries into standard service frameworks, secure defaults became the norm across major languages.

Advanced TLS features, like hot-reloadable SSL contexts for `java-grpc` and `java-jetty`, allow applications to use renewed certificates without restarts, enhancing operational resilience.

Improving Security at Scale

Scaling cert-manager at LinkedIn revealed challenges with default configurations under high load, particularly during "deployment churn" where significant pod restarts occur.

The system must support thousands of nodes and hundreds of thousands of pods per cluster, demanding optimized performance from cert-manager components.