# LinkedIn Overhauls Log Storage _LinkedIn replaces its Kafka log storage with Northguard and Xinfra for improved scalability and operability to handle over 1.2 billion members._ **Updated:** 2026-08-22 **Published:** 2026-05-22 **Source:** https://www.startuphub.ai/ai-news/technology/2026/linkedin-overhauls-log-storage --- LinkedIn is phasing out its long-standing Kafka infrastructure for a new homegrown system dubbed Northguard. This overhaul aims to address the exponential growth in data volume and complexity since Kafka was first implemented 15 years ago. The move is detailed in a recent [LinkedIn Engineering](https://www.linkedin.com/blog/engineering/infrastructure/introducing-northguard-and-xinfra) post. LinkedIn's GrowthDriverover 1.2 billion members and exponential data volume increaseFrom the article 8 mentionsThis overhaul aims to address the exponential growth in data volume and complexity since Kafka was first implemented 15 years ago.Data ReprocessingContextcrucial for debugging and verification of data streamsFrom the article 7 mentionsData streams are fundamental to LinkedIn's thousands of services, enabling them to subscribe to and process information from other services.leads toKafka ChallengesDrivermetadata bottlenecks, cluster size, load balancing difficultiesFrom the article 6 mentionsLinkedIn is phasing out its long-standing Kafka infrastructure for a new homegrown system dubbed Northguard.necessitatesIntroducing NorthguardCoreLinkedIn's new homegrown log storage systemFrom the article 7 mentionsNorthguard is engineered for high scalability and operability.Xinfra LayerCorepart of the new log storage infrastructureFrom the article 4 mentionsTo ease the transition from Kafka, LinkedIn developed Xinfra, a virtualized Pub/Sub layer that sits above Northguard.Enhanced OperabilityEffectaddressing challenges of the previous Kafka systemFrom the article 2 mentionsNorthguard is engineered for high scalability and operability.supportsImproved ScalabilityEffecthandling 32 trillion records daily across 17 petabytesFrom the article 2 mentionsThis abstraction layer maintains compatibility with existing applications while leveraging Northguard's improved scalability and operability. Data streams are fundamental to LinkedIn's thousands of services, enabling them to subscribe to and process information from other services. The ability to reprocess data is crucial for debugging and verification. Kafka, developed by LinkedIn a decade and a half ago, became the backbone for this ordered data pipeline, known as a log, supporting everything from user activity to AI features. However, scaling Kafka to accommodate over 1.2 billion members has proven increasingly challenging. The platform now handles 32 trillion records daily across 17 petabytes, distributed across 10,000+ machines. Key issues included metadata and cluster size bottlenecks, load balancing difficulties, and compromises on consistency in favor of availability. ## Introducing Northguard Northguard is engineered for high scalability and operability. It shards data and metadata, minimizes global state, and employs a decentralized group membership protocol. Log striping is central to its design, ensuring even load distribution by breaking logs into smaller, replicated chunks called segments and ranges. The system's data model consists of records, segments (sequences of records, serving as the unit of replication), ranges (contiguous key space segments), and topics (named collections of ranges). This structure allows for granular control and efficient replication. Northguard's metadata is managed by vnodes, which are fault-tolerant replicated state machines utilizing Raft. A coordinator, the leader of a vnode, handles metadata operations and persists state, enabling seamless leader transitions. This metadata is sharded across vnodes using consistent hashing via a Dynamically-Sharded Replicated State Machine (DS-RSM) to prevent hotspots. ## The Xinfra Layer To ease the transition from Kafka, LinkedIn developed Xinfra, a virtualized Pub/Sub layer that sits above Northguard. This abstraction layer maintains compatibility with existing applications while leveraging Northguard's improved scalability and operability. This evolution signifies LinkedIn's ongoing commitment to robust distributed systems log management, building on foundational technologies like Kafka to support its massive scale. The development of Northguard and Xinfra represents a significant step in ensuring the platform's future infrastructure can keep pace with its user growth and data demands, similar to how other platforms are innovating in [distributed systems](/ai-news/technology/2026/snowflake-kafka-connector-v4-arrives). The adoption of Northguard and Xinfra underscores the critical role of scalable log storage systems in modern, data-intensive platforms, pushing the boundaries of what's possible in areas like [AI development](/ai-news/tech/2026/linkedin-s-ai-memory-platform). --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory. © StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.