● Covered by 1 source · 1 reportMedium impact1 neutral
LLMs are more than just next-token predictors due to post-training with RLVR
Key points
- LLMs are trained autoregressively to predict the next token.
- Pre-training makes tokens more likely if they appeared in training data.
- Post-training with RLVR enables LLMs to explore and learn from new sequences.
- RLVR makes tokens more likely based on high reward outcomes, not just data presence.
The Basic Premise of Next-Token Prediction
The statement that "LLMs are next-token predictors" is a foundational, though incomplete, description of their operation. Transformer-based language models generate tokens autoregressively, meaning they predict the next token based on the preceding sequence. During pre-training, the model's objective is to make the actual next token in a given sequence more likely to be sampled, based on its occurrence in the training data.
Pre-training: Learning from Existing Data
In the pre-training phase, an LLM learns by repeatedly observing sequences of tokens from a vast dataset. For each position in a sequence, the model is trained to predict the token that actually followed the prior tokens. This process involves loss functions, gradients, and parameter updates that collectively increase the probability of the observed next token. Essentially, the base model learns to replicate patterns and sequences present in its training data.
Post-training and the Role of RLVR
Modern LLMs are not just base models; they undergo significant post-training. A key component of this post-training is Reinforcement Learning with Verifiable Rewards (RLVR). Unlike pre-training, where the model only learns from existing sequences, RLVR allows the model to explore by generating new sequences and evaluating their outcomes. This exploration enables the model to learn from novel interactions and feedback.
How RLVR Changes Token Likelihood
During RLVR, the mechanism of making a token more likely operates differently. Instead of increasing the probability of a token because it appeared in the training data, RLVR increases the probability of an explored token because the sequence containing it earned a high reward. This fundamental shift means that post-trained LLMs, while still exhibiting the structure of a next-token predictor, are driven by a different learning objective that allows for more complex and goal-oriented behavior.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
The common understanding that Large Language Models (LLMs) are merely next-token predictors is incomplete, as it only describes their pre-training phase. Post-training, particularly with Reinforcement Learning with Verifiable Rewards (RLVR), allows LLMs to generate new sequences and learn from their outcomes, moving beyond simply replicating existing data. This distinction is crucial for understanding the capabilities and development of modern LLMs.