LinkedIn Overhauls Log Storage

LinkedIn is phasing out its long-standing Kafka infrastructure for a new homegrown system dubbed Northguard. This overhaul aims to address the exponential growth in data volume and complexity since Kafka was first implemented 15 years ago. The move is detailed in a recent LinkedIn Engineering post.

Data streams are fundamental to LinkedIn's thousands of services, enabling them to subscribe to and process information from other services. The ability to reprocess data is crucial for debugging and verification. Kafka, developed by LinkedIn a decade and a half ago, became the backbone for this ordered data pipeline, known as a log, supporting everything from user activity to AI features.

However, scaling Kafka to accommodate over 1.2 billion members has proven increasingly challenging. The platform now handles 32 trillion records daily across 17 petabytes, distributed across 10,000+ machines. Key issues included metadata and cluster size bottlenecks, load balancing difficulties, and compromises on consistency in favor of availability.

Introducing Northguard

Northguard is engineered for high scalability and operability. It shards data and metadata, minimizes global state, and employs a decentralized group membership protocol. Log striping is central to its design, ensuring even load distribution by breaking logs into smaller, replicated chunks called segments and ranges.

The system's data model consists of records, segments (sequences of records, serving as the unit of replication), ranges (contiguous key space segments), and topics (named collections of ranges). This structure allows for granular control and efficient replication.

Northguard's metadata is managed by vnodes, which are fault-tolerant replicated state machines utilizing Raft. A coordinator, the leader of a vnode, handles metadata operations and persists state, enabling seamless leader transitions. This metadata is sharded across vnodes using consistent hashing via a Dynamically-Sharded Replicated State Machine (DS-RSM) to prevent hotspots.

The Xinfra Layer

To ease the transition from Kafka, LinkedIn developed Xinfra, a virtualized Pub/Sub layer that sits above Northguard. This abstraction layer maintains compatibility with existing applications while leveraging Northguard's improved scalability and operability.

This evolution signifies LinkedIn's ongoing commitment to robust distributed systems log management, building on foundational technologies like Kafka to support its massive scale. The development of Northguard and Xinfra represents a significant step in ensuring the platform's future infrastructure can keep pace with its user growth and data demands, similar to how other platforms are innovating in distributed systems.

The adoption of Northguard and Xinfra underscores the critical role of scalable log storage systems in modern, data-intensive platforms, pushing the boundaries of what's possible in areas like AI development.