A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo
Wells Fargo pre-trained a transformer from scratch on billions of customer transactions rather than words, then fine-tuned it for fraud detection, credit decisions, anti-money laundering and marketing. The talk covers how tabular transaction data was tokenized, the model's size and training setup, and results that beat the production XGBoost fraud model on both detection and false positives while meeting real-time latency requirements.
Why transactions need their own foundation model
Language foundation models learn by predicting the next token in text, then go through supervised fine-tuning and RLHF or RLVR-style post-training. Financial services data is different. Much of it is tabular, spanning customer records in consumer and investment banking, and a large share is transaction history from debit cards, credit cards and digital checks across many channels. Feeding that into a language model can work, but it is not the best fit. Wells Fargo’s bet is a single transaction backbone that learns representations of customer behavior at scale and can then be reused for credit line increases, fraud detection, anti-money laundering and marketing, instead of building a separate model for each task.
Scale and the five dimensions
The data volume is large: billions of transactions per month, about a million accounts, and hundreds of transactions per client each month. Rather than hand-crafting features for each downstream task, the team went back to the raw features and reduced them to roughly 100 core information units. Thousands of engineered features already existed, but the aim was to isolate the irreducible signal and let the architecture learn the interactions. These raw features fall into five dimensions:
- Who: the customer or the merchant involved
- What: what is being bought or transacted
- How: the channel, whether credit, debit or digital
- When: timing of the transaction
- Where: the location
Merging all digital, checking and savings activity still leaves a limited, highly specialized vocabulary. That is one reason existing pre-trained models were not a good starting point.
Turning tables into tokens
The hardest engineering problem is the vocabulary. The data mixes categorical fields with very different cardinalities, from fields with only eight values to identifiers like IP addresses and postal codes. It also includes numerical quantities such as transaction amount and the time elapsed between transactions. Encoding every zip code as its own token would make the vocabulary too large to train well, so the real work was deciding which information goes into each token and how many tokens each characteristic deserves. Zip codes, for instance, can be regrouped according to their distributions rather than treated as isolated IDs. Each incoming transaction is then mapped through a lookup to token IDs and embeddings. The team considers this tokenization, together with post-training, to be the real secret sauce.
Architecture and training
The model is a decoder-only transformer pre-trained from scratch. No open weights were used. The team took architectural guidance from existing open tabular foundation models, then trained on its own GPU cluster over days to weeks. The best-performing size is 1 billion parameters, which gave the strongest balance of cost and gain, and an 11 billion parameter version is in development. This is far smaller than frontier language models, which fits a narrower domain.
Scaling followed Chinchilla-style reasoning, but the initial heuristic of about 20 training tokens per parameter was not enough for this data. Experiments showed the model needed much more data per parameter than the standard recipe suggests. The core training objective is next-transaction prediction, with the usual language-modeling objectives as reference points. The main training decision is how much transaction history goes into each context window. Evaluation tracked perplexity, accuracy and downstream task dashboards.
Efficiency came from standard practice: mixed-precision training, distributed data parallelism, the largest micro-batch size the hardware allowed for faster convergence, RoPE positional encoding, and FlashAttention with activation checkpointing for memory efficiency.
Fraud detection results
Fraud was the first downstream task. The production model is an XGBoost classifier trained on labeled fraudulent and legitimate transactions. Its blind spot is the relationship between the transaction being scored and the customer’s full history, which is hard to capture through hand-built features. Transformer attention links the current transaction directly to past transactions, and the team sees this as the main reason detection improved. At inference time the input is the customer’s transaction sequence, and the question becomes whether this new transaction is fraudulent given everything that came before.
The metrics that matter map directly to dollars: detection rate and false positive rate. Pushing detection higher usually increases false positives, but here the model caught more fraud and produced fewer false alarms at the same time. False positives carry real cost, because a wrongly declined card triggers customer calls, operational work and lost legitimate sales. Latency was also a hard constraint, since card declines must be decided within 250 milliseconds. The model ran roughly ten times under that budget. Against the XGBoost baseline, the team reports savings in the millions of dollars.
One backbone, many tasks
The central result is a reusable backbone. Once pre-trained, the model predicts the next transaction well, and fraud detection was the first task fine-tuned on top of it. Credit line increases, anti-money laundering and marketing are now under investigation on the same foundation. The longer-term goal is for different teams across the organization to draw on one set of learned customer representations rather than maintaining a separate model for each use case.