Each layer applies an affine map followed by a nonlinear activation, so the network composes functions rather than choosing from a hand-built basis. Without the nonlinearity the whole stack collapses back into one linear map.

Calling it 'hidden' only means unobserved: the layer's values are intermediate computations. Each coordinate is a learned feature detector, which is precisely what distinguishes an MLP from basis expansion.

Without it, composing linear maps yields one linear map. Which activation to pick is a gradient question — the derivative decides whether signal survives a deep stack — so the choice is argued in the training-stability module, not here.

Piecewise linear with two pieces, so a ReLU network is a piecewise-linear function — which is what lets region counting measure expressiveness. Its derivative is exactly one on the active side and zero on the dead side.

An existence theorem, not a recipe: it says nothing about the width required, whether gradient descent finds the approximant, or whether it generalises. The constructive intuition is that sums of ramps make any piecewise-linear interpolant.

Each ReLU unit carves space with a hyperplane; regions are the pieces of the resulting arrangement. Counting regions turns 'what can this network express' into combinatorics, which is what makes depth-versus-width comparisons provable.

One hidden layer's region count grows polynomially in width, while composing layers folds space so counts multiply — exponential in depth. Telgarsky's sawtooth gives an explicit function separating the regimes.

Identity for a Gaussian mean, sigmoid for Bernoulli, softmax for categorical. The paired gradient with respect to the pre-activation never carries a saturating activation derivative, so the last layer cannot kill its own training signal.

Softmax couples the logits into a distribution over mutually exclusive classes. A multilabel problem is a stack of independent Bernoulli decisions — one sigmoid each — because 'cat' and 'indoors' are not competitors.

Sigmoid with squared error is the canonical mistake: the gradient carries a σ'(z) factor that vanishes when the unit saturates, so a confidently wrong prediction barely moves the weights.

Each layer stores its forward output and multiplies the incoming adjoint by its local Jacobian. The naive version recomputes shared suffixes exponentially often; caching them is the entire trick.

Not numerical differentiation (finite differences, O(√ε) error) and not symbolic differentiation (expression swell). Reverse mode accumulates gradients backward through a tape; it is what every deep-learning framework implements.

Keeping only boundary activations between segments drops activation memory from O(L) to O(√L) at the price of roughly one extra forward pass. Standard on any run where the activation ledger is the binding constraint.

A product of L matrices scales like σ^L: below 1 it vanishes exponentially, above 1 it explodes. Fixes attack the multiplier — activation choice, initialisation scale, clipping, residual paths, normalisation.

He initialisation draws weights with variance 2/d_in; Xavier uses 1/d_in (or 2/(d_in+d_out)). The wrong constant compounds by depth — too large explodes, too small vanishes — before learning even starts.

Norm clipping projects onto the Euclidean ball ‖g‖ ≤ c; per-value clipping onto the cube, which rotates the direction and leaks a √N factor. The threshold is read off the run's own norm histogram, not theory.

The Jacobian of a residual block is I + J', so the backward pass keeps a term of norm one no matter how deep the stack. The network learns the perturbation to add rather than the whole map — the difference between fitting and re-representing.

The same example normalises differently depending on its companions, which couples training examples, breaks at batch size 1, and forces running statistics at inference. Per-example normalisers (layer norm, RMSNorm) remove that coupling entirely.

Layer norm subtracts the mean and divides by the per-example standard deviation; RMSNorm skips the mean entirely and divides by the root mean square. Both keep activations on a fixed-radius sphere; modern transformers overwhelmingly run RMSNorm.

Post-norm wraps the residual (Norm(x + Sublayer(x))) and needs warmup plus careful init; pre-norm puts the norm inside the branch (x + Sublayer(Norm(x))) so the trunk stays a bare sum that gradients cross unscaled — at the price of a residual stream whose variance grows with depth.

In high dimension, most critical points of a random-function landscape are saddles: the chance that all d eigenvalues of the Hessian come out positive shrinks exponentially in d. The optimiser's real work is escaping plateaus, not rolling downhill.

Heavy-ball momentum accumulates v ← βv + g and steps along −ηv; Nesterov evaluates the gradient at the lookahead point. Along a valley it accelerates descent by roughly 1/(1−β); across it, perpendicular wiggles average out.

m tracks the mean gradient, v the uncentred second moment; the update m̂/(√v̂ + ε) makes each parameter's step scale-free. The default optimiser for transformers, at the cost of 2 extra floats per parameter of optimiser state.

Adding λθ to the gradient makes Adam's per-coordinate scaling shrink the decay too, so effective regularisation varied with gradient scale. AdamW decays θ by ηλθ outside the adaptive term, which is why it generalises better at the same training loss.

At initialisation the adaptive denominators and post-norm residual stacks are both unreliable; a linear warmup smuggles the model past the phase where one bad step can bias the whole trajectory.

Past the point where the model just fits the training set, adding parameters finds smoother interpolants, and generalisation recovers. It reframes 'bigger overfits' as a statement that only holds on the under-parameterised side.

The classical example is separable logistic regression, where SGD diverges toward the max-margin separator. In deep nets the bias shows up as a preference for low-curvature solutions, which is part of why over-parameterised nets generalise at all.

At inference the layer keeps all units but scales them (or divides at train time) so expectations match. The regularisation is architectural: it forbids any feature from leaning on the exact value of another.

The cross-entropy minimum is unbounded for a hard one-hot target; smoothing caps the achievable confidence, which regularises the logit scale and usually improves calibration — while also changing what a distilled student can learn from the softened teacher.

The stopped step count plays the role of a capacity knob: parameters see only as much data-fitting as the stopped trajectory reached. It doubles as the implicit prior that later steps are less trustworthy than earlier ones.

The augmentations encode the domain's known invariances, which is cheaper than learning them. Mixup and CutMix go further: they interpolate inputs and labels, changing the objective itself rather than just expanding the support.

The layer's output is a sliding dot product — the same template matched everywhere — so a shift in the input produces a shifted copy of the feature map (equivariance), and the parameter count no longer scales with image size.

Each layer widens the field by roughly the kernel size times the running stride product. A classification head needs a field at least the object's size; dilated kernels grow the field without paying resolution.

Max pooling keeps the strongest response in each window; strided convolution achieves the same downsampling with learned filters. Modern architectures mostly prefer the latter, since the stride buys invariance without throwing away magnitudes.

Early layers learn general detectors — edges, textures — that transfer across tasks. Freezing the trunk and training a linear probe is the cheap end; full fine-tuning the expensive end. The whole lesson is deciding where between them to land.

Semantically it is E^T t; computationally it is a gather. The gradient touches only the rows that were looked up, which is why the layer is cheap while a dense equivalent would stream the whole table per token.

The bottleneck forces the network to keep what the reconstruction needs. As a generator it fails — nothing makes the latent space samplable — which is exactly the gap the VAE fills with a prior and a variational posterior.

A pretext task is useful only if solving it requires real structure. Masked prediction (BERT, MAE) forces the model to learn syntax and semantics; trivial pretexts collapse to shortcuts that transfer nowhere.

InfoNCE is softmax cross-entropy over one positive and N−1 negatives; its optimum lower-bounds mutual information between the views. Temperature controls how sharp the softmax is, and batch size controls how many negatives push back.

The same weights apply at every step, so the model handles any length — but the gradient must cross every step (BPTT), the hidden state is a bottleneck the whole history compresses into, and the loop is sequential: no parallelism across time.

The LSTM's cell update is a sum controlled by gates rather than a repeated product, which is what lets credit cross hundreds of steps. The GRU merges the gates to three and drops the separate cell — usually equivalent in practice.

Under teacher forcing the model never sees a prefix containing its own mistake; at generation, one wrong token shifts the input off the training distribution and distributions after that are guesses. Scheduled sampling mixes the two during training.

Beam search finds higher-probability sequences than greedy decoding but still approximates the mode, and it trends toward short, generic output. For open-ended text it lost to sampling; it survives where there is a right answer (translation, speech).

The 1/√d_k scaling keeps logits out of softmax's saturating tails as dimension grows. One step of attention does what a recurrent layer takes T steps to do — read distant context — at the price of a T×T score matrix.

With a mask, one forward pass computes the prediction for every position simultaneously; without it the model would need a pass per token or peek at the answer. At inference the mask is inherent — the future simply does not exist yet.

Heads are not redundancy: a single softmax per position can express only one attention pattern, and h smaller heads cost the same FLOPs as one big one. Empirically heads specialise — some track position, some content, many are prunable.

The score depends on i−j automatically because rotations compose: R_i R_j^⊤ = R_{i−j}. Frequencies span geometric scales so some dimensions track local position and others long range; extrapolating past the trained length needs the rope-extension tricks the serving module covers.

Each head gets a different slope m_h from a geometric sequence, so short-range heads stay local and long-range heads barely decay. Because the bias is a function of offset rather than of absolute position, ALiBi extrapolates to longer contexts than it trained on.

Per token the cache reads T·d·layers·2 values — decode is memory-bandwidth-bound on the cache, not compute-bound. Its size grows linearly in context length and batch, which is why paging, quantisation and head-sharing (GQA/MLA) exist.

GQA gives each group of query heads one shared K,V pair — the cache divides by the group size. MLA goes further: it stores a down-projected latent per position and re-expands on read, the DeepSeek trick that cuts the cache by an order of magnitude.

Each multiply-add counts twice (multiply, then add) once per parameter per token; the backward pass costs two forwards. Attention adds a term ∝ L·T·d that matters at long context. FLOPs are what the hardware sells; wall-clock is what memory traffic charges.

FLOPs per byte read decides the regime: below the ridge the kernel waits on memory, above it the ALUs saturate. Decode-time attention is famously bandwidth-bound — one token, a whole cache read — which is why batching helps it so much.

bf16 keeps fp32's 8 exponent bits and sacrifices mantissa — safe for activations and weights, bad for accumulators. fp8 halves again and works only with per-tensor scaling plus an fp32 master path, since its range saturates at ±448.

The kernel keeps running softmax statistics (max and normaliser) per tile and rescales on the fly, reading K,V once instead of writing and re-reading the full attention matrix. Same math, an order of magnitude less memory traffic — the reason it replaced every other 'efficient attention' in practice.

Dropping the softmax lets the K^⊤V outer products accumulate into a d×d matrix state updated recurrently. Exact at prediction but weak at exact recall: the state must compress all history into a fixed matrix, which is why hybrids keep a few full-attention layers.

h'(t) = Ah(t) + Bx(t) becomes a scan h_t = Ā_t h_{t−1} + B̄_t x_t. Mamba's selection makes Ā,B̄ functions of the input, so the state can gate what to keep — expressiveness where plain linear recurrence fails, at O(1) memory per step.

A softmax router picks top-k experts per token; the model's capacity lives in total parameters while its cost lives in active parameters. The catch is load balancing — a router that favours a few experts wastes the rest, so auxiliary losses or balancing terms keep the routing even.

BPE starts from characters and merges greedily until the vocabulary is full. The result decides what the model can say cheaply: rare words split into many tokens, unseen scripts may split into bytes, and token boundaries leave artefacts the model has to unlearn.

L(N, D) ≈ E + A/N^α + B/D^β fits the training loss across orders of magnitude. Chinchilla's version says optimal training allocates parameters and tokens in a fixed ratio — which is why 'compute-optimal' pins a tokens-per-parameter constant rather than a size.

Data parallelism replicates the model and syncs gradients; tensor parallelism shards matmuls inside a layer across fast interconnects; pipeline parallelism assigns layer ranges to stages and streams microbatches through them. Real runs compose all three — the lesson is which constraint each one relieves.

Plain data parallelism keeps a full copy of everything on every device; ZeRO's three stages shard the optimiser states, then gradients, then weights themselves, gathering each shard only during its use. Communication rises modestly while per-device memory collapses.

Sampling with temperature 0 is greedy decoding; T=1 draws the model's own distribution. The knob trades reliability for variety, and it is load-bearing in distillation, where the same T>1 reveals the teacher's dark knowledge in its softened classes.

Top-k truncates at a fixed count, which either prunes the tail too early or admits junk when the distribution is flat. Top-p adapts: the set grows or shrinks with the distribution's shape, keeping the nucleus at roughly fixed quality.

Prefill runs a full self-attention over the prompt at once; decode appends tokens serially, each reading the whole KV cache. Serving systems split them onto different hardware or batch decode aggressively — a mixed batch serves neither phase well.

Static batching wastes the whole batch's tail while the longest sequence finishes. Continuous batching evicts completed sequences and admits waiting ones every step, which is the main throughput lever of modern serving stacks.

Virtual-memory for the KV cache: instead of reserving max-length contiguous space per sequence, blocks are allocated on demand and shared between sequences with common prefixes. The trick that made long contexts servable at scale.

Verification is a single forward over the drafted tokens, so a match costs the same as one normal token and acceptance rates determine the speedup. The output distribution is unchanged — it is exact acceleration, not an approximation.

Per-channel scales keep each output channel's error independent; the hard part is activation outliers, which is why the course separates weight-only quantization from the harder W×A schemes and from calibration methods that re-fit the rounding.

Soft targets at temperature T>1 expose the teacher's class relationships (cat resembles dog more than car); the student trained on them beats the same student trained on hard labels, even on classes the teacher demonstrably got wrong.

Unstructured magnitude pruning gives zeros a dense kernel cannot skip; 2:4 semi-structured patterns pay only on sparse tensor cores; structured channel removal shrinks the dense matmul itself. Prune, retrain with the mask frozen, measure realised — not nominal — speedup.

A few labelled examples in-context steer behaviour with no gradient update at all. The mechanism is induction over the prefix — the model reads its own demonstrations as evidence — and it is brittle in ways fine-tuning is not.

SFT is where style, refusal and format come from; small high-quality sets outperform large noisy ones (LIMA's claim). The price is catastrophic forgetting — weight updates that fix the behaviour can erode the pretrained capabilities underneath.

The fixes trade memory for stability: replay buffers mix old data back in, EWC penalises moving weights that mattered for the old task, adapters keep the pretrained weights frozen entirely. Continued pretraining shows the same failure at scale.

The frozen base means no optimiser state on the bulk of the model and adapters that swap at inference. QLoRA pushes further: 4-bit-quantized base weights plus a bf16 adapter, fitting large-model fine-tuning onto a single GPU.

A dense retriever scores the query against an indexed corpus; the top passages land in the context as evidence the model conditions on. RAG trades context length for currency — but retrieval failure and context-fitting limits are the lesson's honest caveats.

A prompted scaffold ('think step by step') or a few worked rationales changes what the model computes: intermediate tokens act as working memory. The gains concentrate on tasks with real intermediate structure — arithmetic, symbolic — and shrink on recall tasks.

Self-consistency samples several chains and votes; verifiers rank candidates; reasoning models stretch a single chain. The trade-off the lesson quantifies: when extra tokens beat extra parameters, and when they just buy noise.

Maximising the ELBO teaches the encoder to produce posteriors and the decoder to invert them; the KL term pulls posteriors toward N(0,I) so the latent space becomes samplable — the property a plain autoencoder lacks and GANs get by other means.

Sampling from N(μ,σ) has no gradient with respect to μ,σ; moving ε outside the parameters turns the node into a deterministic transform of a fixed random input. Without it, backpropagating through the VAE's latent is a REINFORCE problem instead of a derivative.

The minimax objective is beautiful and brittle: saturating losses starve G, unbalanced training oscillates, and mode collapse lets G win by memorising a few outputs. Most practical tricks — Wasserstein distances, feature matching — patch the game's dynamics rather than its goal.

The forward process adds Gaussian noise t times until x_t ≈ ε; the model is trained to predict the noise that was added, so sampling runs the chain backward step by step. Slow but stable — the mode-collapse failure GANs live with is absent.

Estimating the score avoids the partition function entirely; denoising score matching does it by predicting noise at multiple scales. The connection runs both ways: diffusion training is score matching, and DDPM sampling is Langevin by another name.

The scale s trades fidelity for diversity: s>1 pushes samples toward the condition and away from the model's generic prior. No separate classifier is needed — the model supplies both endpoints — which is why every modern image model uses it.

The VAE compresses an image into a small latent grid; the denoiser works there and the decoder maps the result back to pixels. Perceptually near-lossless, computationally transformative — the trick that put image generation on a single GPU.

Rather than learning to denoise each step, the model regresses the straight-line velocity field between noise and data; sampling integrates the ODE. Rectified flow's linear interpolation makes near-straight paths, which is why few-step samplers work at all.

Each image and its caption are pulled together while mismatched pairs are pushed apart. The result is zero-shot classification ('a photo of a {class}' competes for the image's embedding) and the text encoder every image generator steers by.

It is the exponentiated average cross-entropy, so it measures next-token fit, not quality — a model can memorise a benchmark and look brilliant. Tokenisation changes the denominator, which is why bits-per-byte exists as the comparable alternative.

Detection runs from n-gram overlap checks to probing whether the model can reproduce held-out examples verbatim. The deeper version is near-duplicates and paraphrases that exact matching misses; contamination-aware reporting is now part of an honest eval.

Judges inherit systematic preferences: longer answers score higher, a model prefers outputs resembling its own style, and order effects flip pairwise outcomes. The defence is protocol — position swaps, rubric anchoring, and human calibration on a sample.

The mechanism is statistical: training rewards confident completions and benchmarks reward guesses over abstention, so the model has no channel for 'I don't know'. Retrieval and calibrated sampling mitigate; the failure is structural, not a bug version-by-version.

Expected calibration error bins predictions by confidence and compares to hit rate; modern LLMs are overconfident in ways RLHF made worse. Post-hoc fixes — temperature scaling on held-out data — repair the probabilities without touching the ranking.

Any retrieved or pasted text can carry instructions the model will follow; grounding expands the attack surface rather than removing it. Defences are architectural — least privilege, separation of trusted instructions from data — not prompt engineering.

Basis expansion fixes the features before seeing data; an MLP parameterises the features themselves and lets gradient descent shape them. XOR is the two-point proof that no fixed linear boundary suffices.

The pair is chosen so the last layer cannot saturate its own signal. Deliberately mismatched pairs (sigmoid with squared error) produce a σ'(z) factor that goes to zero exactly when the prediction is most wrong.

Softmax forces the logits to compete — right when one answer exists, wrong when 'cat' and 'indoors' are both true. The choice is about the semantics of the label space, not the loss's convenience.

Universal approximation already lets one wide layer represent anything; the argument for depth is efficiency, not possibility. Telgarsky's sawtooth is the explicit witness: 2^L linear pieces from L layers versus exponential width for a shallow copy.

Read it as 'the model class is not the bottleneck', then look to optimisation and sample complexity for the guarantees actually doing work.

Gradients vanish or explode multiplicatively with depth. Variance-preserving init (He for ReLU, Xavier for tanh) keeps forward activations and backward gradients at unit scale; everything downstream assumes it.

Clipping by norm caps the worst-case step without re-scaling every batch the way clipping by value does. The threshold sits just above the healthy median norm, read off a training curve.

Skip connections turn a depth-L product of Jacobians into a sum of near-identity terms, so the loss surface stays connected and early layers keep receiving gradient even at init.

BatchNorm borrows statistics across the batch — which couples examples at train time and breaks under small or distributed batches. LayerNorm normalises within one example, so it behaves identically in training and inference.

Pre-norm keeps an identity gradient highway and trains stably at depth but can under-represent late layers; post-norm recovers the sharper fit at the cost of warmup and fragility.

SGD+momentum still wins where you can afford to tune it, but Adam-family defaults dominate because they rescale coordinates automatically. Coupling decay to the adaptive gradient silently shrinks weights more where gradients are large — decoupling is the fix.

Early steps need a small LR while second-moment estimates and representations settle; late training needs decay to land in a wide basin. Constant-high LR is the schedule that fails for a measurable reason.

The gradient-noise scale marks where extra samples stop adding information per step. Small-batch noise is itself a regulariser, so very large batches can need retuning of decay and epochs, not just LR.

A curve is a diagnosis, not a score — train-vs-val divergence, update-to-parameter ratio, and gradient-norm spikes each point at a different knob (regularisation, LR, clipping, init).

Augmentation encodes invariances the model would otherwise spend parameters learning. Over-parameterised nets can still interpolate and generalise (double descent), so capacity is rarely the first thing to cut.

A dense layer relearns every pattern at every position; a kernel shares it. Pooling and stride then buy receptive field cheaply.

Pretrained features are generic near the input and task-specific near the output; the lower the target task's data, the more of the network stays frozen.

The pretext's job is to make representation quality, not the invented task, the bottleneck. Contrastive objectives pull augmentations together and push other examples apart; masked reconstruction learns from what was removed.

Embeddings turn a discrete index into a learned geometry: distances and directions become trainable. The table scales with vocabulary, not with input length.

Compressing the whole input into one state is what fails as inputs lengthen; attention-as-alignment lets each output step read the relevant source positions directly.

Attention is memory-bandwidth-bound, not FLOP-bound: materialising the N×N matrix is the real cost. When approximation is needed, sliding-window preserves local structure while linear attention trades the softmax kernel for a recurrent form.

SSMs compress the past into a fixed state like an RNN but train like a convnet; the selection mechanism restores content-based routing. Hybrids get most of the efficiency while keeping exact-retrieval paths.

The cost moves to routing: load balancing is an auxiliary-loss problem, and expert parallelism adds all-to-all communication. MoE is a systems trade as much as an architecture one.

Loss falls as a power law in compute, data and parameters; Chinchilla's point is that models were under-trained, not under-sized. Beyond the frontier the trade-offs shift to data quality and inference cost.

Deduplication removes memorised boilerplate, quality filters pass classifiers over documents, and decontamination strips benchmark overlap so evals keep meaning something.

The decision is driven by the memory budget — parameters, gradients, optimizer states, activations — then by interconnect: tensor parallelism wants fast intra-node links, pipeline tolerates slower ones at the price of bubbles.

bf16 halves the activations and weights traffic at a small quality cost managed by loss scaling or master weights. Checkpointing trades ~30% more compute for an order less activation memory.

Autoregressive decode is memory-bound, so a small drafter's tokens verified in parallel against the target model give the same output distribution at a fraction of the latency. The KV cache is the prerequisite: no recompute of the prefix.

Request-level batching strands the GPU on the longest sequence; iteration-level scheduling admits new requests mid-flight. Paging removes the contiguous-allocation waste that limited batch occupancy.

Per-channel weight-only PTQ is nearly free at int8; lower bits need calibration to place rounding error where it is cheap. Distillation transfers the function to a smaller student via soft targets; pruning removes weights that saliency says do not matter.

Full fine-tuning costs a full copy of optimizer state and risks forgetting; a rank-r update (r ≈ 8–64) captures most of the task delta at a fraction of trainable parameters. QLoRA quantizes the frozen base further.

Gradient descent optimises the current distribution only; earlier-task performance drops as weights move. Replay mixes old data back in; parameter-efficient methods bound the drift by construction.

RAG keeps the knowledge editable and attributable; the trade is retrieval quality and context budget. Long context can subsume small corpora but pays latency per query and still degrades in the middle.

Needle tests flatter long context — real multi-hop use degrades with distance and cost grows quadratically. RAG's failure mode is recall; long context's is price and mid-context dilution.

CoT externalises intermediate state the forward pass would have to hold implicitly; sampling-and-verifying buys majority reliability where a verifier exists. The trade is monotone in cost, so the question is where the extra tokens pay.

Score matching trains the denoiser by regression against the noised sample — stable where GANs oscillate. Latent diffusion moves the iterative loop to a compressed space, buying quality per step.

Two forward passes (prompted and un) define a direction; scaling it trades diversity for prompt adherence. The same trick replaces the discriminator that guided diffusion once needed.

The InfoNCE objective turns caption pairing into classification over the batch. The resulting shared space gives zero-shot classification, retrieval, and the text encoder most VLMs start from.

A static suite drifts toward the training corpus and saturates; pairwise human/LLM preference tracks perceived quality but rewards style. Report contamination handling before comparing numbers.

Position bias, verbosity bias and self-preference are measurable, not hypothetical. The judge is a metric to be validated against humans on a sample, not a ground truth.

If 'I don't know' scores zero and a guess scores nonzero, the optimum is to bluff. Calibration and selective abstention are the measurable handles — perplexity and RLHF shape the rest.

The model cannot reliably rank instructions from a data channel it was trained to obey. Defences reduce attack surface — they do not restore a security boundary that never existed in the weights.

A soft target carries a whole distribution per example instead of one index; temperature lifts the tail into gradient range and the T² factor keeps the soft loss's contribution stable across temperatures.