AI Engineer host Freddy Lecue, who leads AI at Wells Fargo, said the bank pre-trained its first foundation model from scratch on billions of monthly transactions to replace one-off classifiers. The decoder-only transformer has about 1 billion parameters and was trained for days to weeks on the bank's GPU cluster with zero weights borrowed from open models, Lecue said.
It beats the production XGBoost baseline on both fronts that matter for cards.
Lecue said the new backbone raises fraud detection rate while cutting false positives, and runs well under the 250 millisecond decline window, roughly 10 times faster than the budget requires. The bank started from thousands of hand-engineered features and distilled them to about 100 raw features across five dimensions, who, what, how, when and where, then let transformer blocks learn relationships across a customer's sequence. Vocabulary was the hard part, Lecue said, with heterogeneous fields like transaction amount, delta time, IP address and postal codes, he cited about 2,800 zip-code tokens after grouping, mapped through value-to-token, tokenization and embedding lookup. Architecturally the team used Chinchilla scaling math, initially targeting 20 tokens per parameter and ending closer to 10 after experiments, with mixed precision, distributed data parallel, flash attention and activation checkpointing. A next version at 11 billion parameters is now in work, Lecue said, because 1 billion was the smallest that delivered the clearest ROI.
The immediate deployment is fraud classification. Lecue framed it as sequence conditioning, given a stream of debit, credit and digital check transactions from millions of accounts with hundreds of events per client per month, the model predicts whether the next transaction is fraudulent given history. That attention over past transactions is what XGBoost struggles to encode without brittle manual features, he argued. Lecue tied the metrics directly to dollars, fewer false declines means fewer call-center contacts and fewer lost sales, which he quantified as millions in savings versus the XGBoost state of the art, without publishing exact precision-recall curves or a time-split evaluation window.
On tabular foundations, the bank did not chase a general language model for structured data. Lecue said merging checking, savings and card streams still yields a limited but bank-specific vocabulary that does not transfer well from open pre-trained models, so the team tanked on pre-training to predict the next token or next transaction purely on its own data. That contrasts with recent research directions such as Factorized Tabular Foundation Models, which keep cell representations while factorizing computation through one latent per row and one per column, per the arXiv paper.
The gap a risk team will want closed is generalization. Lecue said the same backbone will be tried for credit line increase, anti-money laundering and marketing, but the talk provided no holdout by geography, merchant category or time period, no drift monitoring and no adversarial robustness. If the false positive improvement holds out-of-time and across new merchants, the budget question flips from whether to fund GPU pre-training to how many downstream teams can reuse one backbone without rebuilding features. Until those cuts are public, the open question is whether the lift is a fraud-specific win or a reusable transaction representation.
