Post-training is a stack, not a single technique.

Post-training hides several different decisions.

A pretrained model has broad statistical capability but no complete assistant contract. It does not inherently know which user intent to prioritize, when to call a tool, what an organization considers safe, or which evidence proves that a long workflow is finished. Post-training supplies those decisions.

There is no standards body that defines one exhaustive boundary. A practical definition is: deliberate work after base pretraining that changes model weights—or a trained accessory such as a reward model—to produce desired behavior. Under this definition, prompting, retrieval, agent orchestration, and serving optimizations are adjacent systems rather than post-training methods.

This distinction prevents a common category error. LoRA and DPO are not alternatives: LoRA describes which parameters move, while DPO describes the learning objective. DPO can be run with LoRA or with full-parameter fine-tuning.

Real recipes compose multiple stages. Tülu 3 combines supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards. Llama 3 combined supervised fine-tuning, rejection sampling, PPO, and DPO. The recipe is a sequence of signals and tradeoffs, not one magic optimizer.

Define the behavior before choosing the optimizer.

Every post-training program begins with a behavior specification: the tasks and users in scope, correct outputs, tool permissions, refusal boundaries, cost limits, and evidence of completion. The same contract must determine data collection and evaluation. Otherwise, the training signal and the acceptance test reward different systems.

Build held-out task suites, pairwise human comparisons, deterministic graders, safety tests, and regression sets before the expensive training run. Split by task and semantic cluster rather than by individual response so near-duplicate prompts do not leak into evaluation.

Change the represented distribution before teaching the assistant.

Continued pretraining applies the language-model objective to additional unlabeled tokens. It is useful for specialized vocabulary, languages, code, scientific literature, or long-context data. It changes what distribution the model represents; it does not directly teach a conversation or tool-use contract.

Domain-adaptive and task-adaptive pretraining demonstrated that further training on representative corpora can improve downstream performance. The boundary is fuzzy: some teams call this post-training, while others call it late pretraining or mid-training.

Protect the general model with replay data and broad regression tests. Continued training on a narrow corpus can improve the target domain while erasing useful behavior elsewhere.

Use demonstrations to establish the assistant contract.

Supervised fine-tuning, or SFT, trains the model to imitate high-quality prompt-response demonstrations. Instruction tuning is SFT over a diverse set of tasks expressed through natural-language instructions. FLAN showed that instruction tuning can improve zero-shot generalization, while InstructGPT used SFT as the starting policy for preference-based reinforcement learning.

SFT is strongest when experts can demonstrate the answer directly. It teaches roles, formatting, tone, tool-call syntax, uncertainty, and task-specific strategies. Data quality, consistency, and coverage matter more than copying every available conversation.

Scale demonstrations only after defining a trustworthy filter.

Synthetic-data pipelines generate prompts, answers, critiques, reasoning traces, or tool trajectories with another model—or with the model being trained. Self-Instruct generates and filters instruction data before fine-tuning. Rejection sampling, best-of-N, majority voting, critique-and-revision, and expert iteration retain samples that satisfy a judge or verifier.

Llama 3.1 used iterative synthetic-data generation, filtering, SFT, rejection sampling, and DPO. The important component is not generation volume; it is whether the selection rule identifies real quality rather than confident style.

Synthetic data scales what the pipeline can already recognize.

Learn relative quality when one perfect answer is hard to write.

Preference data records that one response is better than another. The judgment can come from people, AI judges, rules, or production outcomes. Direct preference methods update the policy from those comparisons without first fitting a separate scalar reward model.

Direct Preference Optimization turns the standard preference problem into a classification-style loss. DPO, IPO, KTO, ORPO, and SimPO belong to the same broad family but make different assumptions about labels, references, margins, and regularization.

Turn repeated judgments into a proxy—then distrust the proxy.

A reward model reads a prompt and response and emits a scalar. It can rank candidates, filter synthetic data, select checkpoints, or provide the training signal for RL. With a Bradley–Terry model, preference probability depends on the difference between the two reward scores.

The reward model generalizes beyond individually labeled examples. That is its value and its central risk. Once the policy is optimized against it, small proxy errors become attractive strategies. Research on reward-model overoptimization shows that proxy reward can continue rising after ground-truth quality begins to fall.

Use online policy optimization when the model must explore.

The canonical RLHF pipeline trains an SFT policy, collects comparisons, fits a reward model, generates fresh policy outputs, and optimizes the policy while constraining drift. PPO is historically prominent, but REINFORCE, RLOO, GRPO, and related policy-gradient methods can play similar roles.

RLAIF changes the source of the comparisons: an AI judge supplies some or all of the feedback. Constitutional AI combines written principles, model-generated critiques and revisions, AI preference labels, and reinforcement learning.

AI feedback can reduce labeling cost. It does not remove the need for human governance, judge audits, or held-out evaluation.

Replace opinion with execution when outcomes can be checked.

Reinforcement learning with verifiable rewards replaces a learned preference proxy with a verifier: unit tests for code, exact or symbolic answers for mathematics, schema validation, successful sandbox execution, or a known environment state.

Tülu 3 applies RLVR after SFT and preference optimization. DeepSeek-R1 showed that large-scale RL can elicit strong reasoning behavior; its RL-only model also produced readability and language-mixing failures, motivating cold-start data and a multi-stage recipe for the final model.

For specialist enterprise agents, RLVR is especially useful when task completion can be checked from real workflow state rather than inferred from a fluent final message.

Train decisions and trajectories when success is delayed.

Outcome rewards ask whether the agent succeeded. Process supervision asks whether intermediate decisions were sound. A training trajectory can contain subgoals, tool selections, arguments, observations, state changes, permission checks, recovery steps, final artifacts, costs, and the evidence of completion.

Process supervision outperformed outcome-only supervision in one mathematical reasoning setting, but that result does not make process rewards universally superior. Step labels can be expensive, noisy, and overly prescriptive.

Tool use can be learned through demonstrations, synthetic traces, preferences, or environmental RL. Toolformer trained models to decide when to call an API, what arguments to provide, and how to incorporate its result.

An agent should learn when to act, when to verify, when to recover, and when to stop.

Safety is a target across methods, not one optimizer.

Safety post-training can use demonstrations, preference comparisons, constitutional feedback, adversarial prompts, red-team data, refusal and safe-completion examples, tool-permission trajectories, targeted editing, and separate input or output classifiers.

Model-level alignment must remain one layer of a system boundary. Sandboxes, least-privilege credentials, confirmation gates, output validation, rate limits, and audit logs constrain what generated text can cause even when the policy makes the wrong decision.

The objective and the parameter update are separate choices.

Full fine-tuning updates all parameters. Parameter-efficient fine-tuning freezes most weights and trains a smaller adapter. LoRA represents updates through low-rank matrices; QLoRA combines low-rank adaptation with a quantized base model.

PEFT is not automatically safer. A small adapter can still introduce broad behavioral regressions and requires the same evaluation discipline.

Transfer behavior into a smaller or cheaper policy.

Distillation trains a student from a stronger teacher’s responses, logits, reasoning traces, preferences, or trajectories. It is both a post-training technique and a deployment strategy: the teacher produces evidence that lets a smaller model approximate useful behavior on a target distribution.

Distillation transfers mistakes as efficiently as competence. The verifier and dataset mixture determine which one dominates.

Compose compatible checkpoints without pretending interference disappears.

Model merging combines compatible fine-tuned checkpoints or task vectors without a full new training job. Methods such as TIES-Merging try to reduce interference between parameter updates. It can be a fast research tool, but capability addition in weight space is not guaranteed.

Turn new evidence into versioned improvement—not silent drift.

Continual post-training uses new failures, preferences, verified production outcomes, or changing task distributions to produce new policy versions. It is the natural destination for an agent that should improve at recurring enterprise workflows.

The loop needs data lineage, consent, replay sets, regression suites, approval gates, and rollback. A survey of continual learning for LLMs emphasizes catastrophic forgetting across continued pretraining and continual fine-tuning.