X has open-sourced the core recommendation system powering its "For You" feed, revealing a profound shift in how the platform determines what users see.
The newly released repository confirms that the feed is now fundamentally driven by a transformer model derived from Elon Musk’s xAI venture, Grok.
This marks the formal end of traditional, heuristic-based social media ranking on one of the world’s largest platforms. The documentation explicitly states that X has "eliminated every single hand-engineered feature and most heuristics from the system." Instead, the Grok-based transformer handles all the heavy lifting, predicting user engagement probabilities based purely on historical action sequences.
The architecture, detailed in the github repository, shows a system orchestrated by the "Home Mixer." This layer pulls content from two primary sources: "Thunder" (in-network posts from followed accounts) and "Phoenix Retrieval" (out-of-network content discovered via ML similarity search).
The real power lies in the "Phoenix Scorer." This component uses a transformer model, ported directly from the Grok-1 open source release, adapted specifically for recommendation system use cases. It takes the user’s entire engagement history—likes, replies, shares, and clicks—and the candidate post, and outputs a suite of probabilities.
Rather than predicting a single "relevance" score, the model predicts the likelihood of a user performing 15 different actions, ranging from positive (P(like), P(repost)) to negative (P(block_author), P(report)). These probabilities are then combined into a weighted score, where negative actions actively push content down the feed.
This reliance on a complex, multi-action transformer model means that the feed is now a pure black box driven by predictive AI. While the code is technically open, understanding *why* a post ranks highly requires understanding the billions of parameters within the Grok-based model, which remains opaque.
The Death of Feature Engineering
For years, social media algorithms relied on thousands of carefully tuned, hand-engineered features—rules like "boost posts with media" or "downrank posts with too many hashtags." This new architecture throws those rules out the window.
The key design decision, according to the documentation, is the complete reliance on the transformer to learn relevance. This simplifies the data pipelines and serving infrastructure, but it also means the platform has outsourced its editorial judgment entirely to an AI model trained on historical user behavior.
This move tightly integrates X’s core product with xAI, creating a direct dependency on Musk’s separate AI company. Every adjustment to the Grok model could now fundamentally alter the experience of hundreds of millions of users on X.
Furthermore, the architecture employs "Candidate Isolation" during ranking. This means the score for a specific post does not depend on which other posts are in the batch. While this makes the scores consistent and cacheable, it also means the AI is scoring content in a vacuum, potentially limiting the model’s ability to optimize for feed diversity beyond the simple "Author Diversity Scorer" applied later in the pipeline.
The open-sourcing effort, which began under the guise of transparency, now reveals a platform that is less transparent than ever before, not due to secrecy, but due to complexity. The algorithm is no longer a set of readable rules; it is a massive, probabilistic AI system.
For users, this means the "For You" feed is now simply a reflection of what the Grok AI believes they are most likely to click on, regardless of whether that content is healthy, diverse, or even accurate. This is the new reality of AI news in social media.
