Written X. A hypothesis must answer for every element of it, so what the records contain — weekday, temperature, tariff plan — decides which questions a rule can even be asked.

Written Y. Its shape names the task before any model is chosen: a real number for regression, a label for classification, an ordering for ranking.

h : X → Y names a rule, not yet a good one — the memorising table and the constant rule are both hypotheses, and the memoriser's perfect record on its own stored rows is the module's first cautionary tale.

A sequence, not a set: written D = {(x_i, y_i)} for i = 1..n, where repeats matter and each draw must share the world's distribution before it may speak for the future.

Both halves can fail — dependence between consecutive rows, or a distribution that drifts over time — and either route breaks the bridge from a sample average to an expectation.

The labels supervise the search: the sample's (x_i, y_i) pairs are exactly what a loss can score a candidate hypothesis against, which is what makes a training error computable.

The tasks change shape — cluster the rows, or model the distribution that produced them — and there is no y to be wrong about, so the loss-and-risk machinery does not apply directly.

On the meter data, predicting tomorrow's kWh — a number whose misses are priced by a continuous loss such as squared or absolute error, not by a count of wrong labels.

On the meter data, predicting whether usage reached 30 kWh — a decision scored by the 0–1 loss, where a near-miss and a wild miss cost identically.

The deliverable is a sorted list — documents for a query, in the Manning text, which frames it as ordinal regression supervised by relevance judgments on query–document pairs.

On the meter data, gathering weekday-like and weekend-like days from temperature and usage alone — unsupervised, so there is no answer key to score against.

The task estimates P(x) — what a typical day looks like — rather than a rule h(x) from inputs to labels; the least supervised of the module's five families.

phi : X → R^d is the contract for what a model may see: two records agreeing on every column are indistinguishable downstream, no matter how different the raw days were.

Three tariff plans become three columns with a single 1 — every level equidistant from every other, where an integer code would assert |A − C| = 2|A − B| for free.

Adjacent levels differ in exactly one coordinate, so the code says where each band sits — warm is at least mild — without claiming the gaps between them are equally sized.

The date factors into day-of-week and day-of-year; MIT's example factors a blood type into group and Rh sign. The parts, not the label, are what a rule may need to weight.

Day-of-week becomes (sin 2πd/7, cos 2πd/7): every day sits the same distance, 2 sin(π/7) ≈ 0.87, from the next — where an integer puts Monday and Sunday maximally far apart.

Temperature × weekend lets the same reading mean different usage on a Saturday; keep the main-effect columns too, so the product modulates rather than replaces them.

The omitted level is read off the constant term, and the ambiguity is real only when a constant is present — the module's map keeps all three plan columns, since nothing there uses one.

ell(g, y) maps one (prediction, truth) pair to a nonnegative number — the per-case bill whose average over the whole distribution is risk.

Doubling a miss quadruples its price, so a few large errors dominate the average — on the three-band table the constant 20 pays 393.5 under it, against 292.5 for the band lookup.

Every kWh of miss costs identically, so the average is gentler on extremes than the square — the same constant pays 15.65 here, in kWh rather than kWh².

Its risk is exactly the probability of being wrong — the error rate — but as a function of the prediction it is flat almost everywhere, so descent methods read no direction; enumerable classes still minimise it directly, which is why the surrogate question is forwarded for the rest.

R(h) = E[ell(h(x), y)] is a property of the triple — hypothesis, distribution, loss — and it cannot be computed, because the distribution is unknown; everything measurable is an estimate of it.

Its answer differs per loss — conditional mean under squared, median under absolute, most probable class under 0–1 — and it needs an unconstrained hypothesis, since a class cannot always take each slice's winner.

The floor is scoped to this input space and this distribution, so 'irreducible' is never unqualified — but a deterministic feature map can only hold it or raise it (merging two bands moves the 0–1 floor from 0.425 to 0.475); lowering it takes information the thinner representation lacked, and never goes below the raw record's own floor.

The optimal rule thresholds the posterior at the cost ratio, not at one half — a missed high day at 5:1 moves the alarm boundary to p > 1/6, at less than half the symmetric rule's expected cost (0.465 against 1.015).

R̂_n(h) = (1/n) Σ ell(h(x_i), y_i) is the training score — unbiased for R(h) only when h was fixed before the data arrived, never when it won the search.

The module's whole learning algorithm: replace the expectation by the sample average and take the argmin over H — what the winning score certifies depends entirely on the class's reach.

Fixed before the data arrive — three constants, twenty-seven band lookups, or the memoriser — and its capacity controls what a low training score certifies about the world's risk: finite size in this module, finer measures when a class is infinite but still learnable.

UML's name for the prejudice that bounds ERM's search — finite in this module's classes — and the only thing standing between the method and a memoriser's worthless perfect score.

In the lesson's tagged-input example, training and test share one joint law. The independent continuous tag lets a memoriser score zero on training rows but match none on fresh draws almost surely; its true risk is the fallback's, with no improvement to the Bayes floor.

E[min_h R̂_n(h)] ≤ min_h R(h): a biased-low estimate of the class's best, and — the clause that saves it from overreach — no bound on how far the deployed rule's own risk sits above its score.

On the eight-draw sample ERM returned h ≡ 10 at true risk 0.62 while h ≡ 50 offered 0.605 — an excess of 0.015 paid for trusting eight rows.

Conditional on the fitted h, its losses are i.i.d. and their mean is unbiased for R(h) — the fixed-hypothesis argument restored by never letting h see the rows.

Spent freely because the test set is not: it exists to compare rules, and any set used to decide has been promoted out of measurement and into the hypothesis class.

ESL's Err_T: a property of the rule in hand, fitted on this training set — not the expected test error, which averages the procedure's risk over the training sets it could have drawn.

ESL's Err: a property of the recipe, not of one fitted rule. A single split is unbiased for it at its training size, but its usual conditional error bar counts only test-row spread. CV refits on overlapping subsets; it does not automatically estimate the missing training-set variance or provide a valid procedure-level interval.

For a 0–1 rate it is √(p̂(1−p̂)/m) — the standard deviation of a mean of m Bernoulli losses — and the quantity the ±1.96·SE error bar is built on.

It need not be the loss the rule was chosen under — an alarm picked at 5:1 can be reported as a plain error rate; the metric must be the one the decision cares about.

With i.i.d. rows and equal folds assigned independently of their values, CV_k is unbiased for expected test error at training size (k−1)n/k. Holdout also estimates expected test error at its own training size; CV reuses every row in both roles. Pessimism relative to full-size training requires expected risk to decrease with training size.

Each fit sees n−1 rows, limiting training-size bias when the learning curve changes little near n. Near-identical training sets do not guarantee positive covariance or higher variance than smaller-k CV; uncertainty depends on the procedure and distribution. The computational cost is n fits.

The eight rows hold two weekend days; a split pocketing both leaves a training set with no weekend at all — stratifying by the flag forces the mix into every fold.

All readings from one household, or consecutive days of one week, belong on one side of every split — a group divided across folds lets the held-out rows' close relatives train the model.

Two mechanisms appear in the lessons so far: preprocessing fitted outside the folds, and dependent rows split across them — ESL's wrong-way example reports 3% where the truth is 50%.

X stacks the rows' feature vectors, so Xw is the vector of fitted scores and X^T X the Gram summary the normal equations invert. The eight-day meter fit produces the 2 × 2 matrix with determinant 2207 — small enough to solve by hand — and a singular X^T X is a bookkeeping failure (exact collinearity or too few rows), not a missing answer.

Two derivations meet here: setting the gradient of (1/n)‖Xw − y‖² to zero, and projecting y onto the columns' span — residual orthogonality X^T(y − Xŵ) = 0 is what makes them the same answer. The closed form costs O(nd² + d³).

SST is what predicting the mean would pay, so R² measures progress over doing nothing — 0.884 for the worked three-rule comparison. Out of sample it can go negative: a stale constant or a shifted day can lose to the mean outright.

μ̂ and σ̂ are estimated, so the transform is a fitted component and belongs inside the fold like any other. The loss's Hessian rescales by the feature scales squared — metres against millimetres stretches the condition number by 10⁶, and descent zigzags accordingly.

When σ_i² is known to differ per row, Gaussian maximum likelihood returns ŵ = (X^T W X)^{-1}X^T W y with W = diag(1/σ_i²): quieter rows count more, in exact proportion to their precision. Laplace noise instead returns the absolute loss — the loss was a noise model all along.

Monomials, steps, hinges and truncated powers all feed the same solver: ŷ = Σ w_j b_j(x) keeps the closed form because the new design matrix Φ simply has different columns — linearity lives in the parameters, not the input. Enough columns (p + 1 = n) pin every training point exactly, which is interpolation, not skill.

E_D[(y − f̂_D(x))²] = σ² + (E_D[f̂_D(x)] − f*(x))² + Var_D[f̂_D(x)] — the degree-15 fit's failure lives in the variance column, the degree-1 fit's in bias², and more data buys the variance term down alone. The trade-off is an accounting identity, not a seesaw — and squared loss is the only loss it splits.

In the SVD basis each direction shrinks by σ_j²/(σ_j² + λ), so df goes from d at λ = 0 toward 0 as λ grows — a smooth complexity knob graded in fractions of a parameter rather than a discrete degree count. λ = σ²/τ² is also the noise-to-prior ratio under the Gaussian-prior reading.

For orthonormal columns the lasso answer is the OLS estimate shrunk toward zero and clipped at it — the subgradient condition |ρ_j| ≤ λ is what lets w_j = 0 be optimal at all. The ℓ₂ analogue never reaches the axis, and the regularisation path it traces as λ sweeps is the sparsity dial.

Fifty identical candidates still produce a winner, and shared validation rows correlate the estimates without removing the effect. The fixes are structural: one untouched test set read once, or nested CV that scores the choosing procedure rather than the chosen model.

The outer scores measure the recipe — choose, then refit — not the model it happened to return, at a price of K_o(K_i|Θ| + 1) + K_f|Θ| + 1 fits (13.08 in the lesson's small worked run). What it does not buy is a verdict on the final candidate itself.

d_i = ℓ_A(i) − ℓ_B(i) cancels the test set's luck because both models faced the same days; SE(d̄) = s_d/√m. A t-test across k-fold scores overstates confidence — the folds' training sets overlap — and statistical significance still is not practical significance.

Agreement cells cancel — the entire difference between two classifiers lives in the days one errs and the other does not. With the continuity correction (|n_A − n_B| − 1)²/(n_A + n_B) it approximates χ²₁; the exact version is a binomial tail.

A grid spends its budget re-sampling the unimportant axis; random buys a hit probability instead — P(at least one top-q hit) = 1 − (1 − q)^T, so T ≥ ln(1 − α)/ln(1 − q) trials reach a top-5% setting with confidence α. Search penalty-like knobs on a log scale, where each decade earns equal trials.

Each round keeps the top share of the field and multiplies its budget — bad trials die cheap and the total spend stays near one full run's. It lies when a candidate's curve crosses late: a slow starter that would have won is eliminated before it shows.

The diagnosis is a shape, not a number: curves converging high together carry the high-bias signature, a persistent gap the high-variance one — the gap prices the estimation error extra data can still buy down. The validation curve's slope prices the next batch before it is bought.

A scaler fitted on all rows prices the fold it should be blind to; a feature computed from the future answers a different question; a column derived from the label is the answer; and one household's days on both sides of a split are one household. The discipline that blocks all four: everything fitted goes inside the fold.

Filters rank a column against the label alone (correlation, mutual information) and miss redundancy; wrappers spend O(d²) fits on forward-stepwise search with CV as the jury; embedded methods — lasso's zeroed weights, a tree's ignored features — arrive free but select for that model only. All three belong inside the fold: screening on all the data first reports pure noise as predictive.

w is the boundary's normal and (w^T x + b)/‖w‖ the signed distance to it; absorbing b as an extra coordinate passes the boundary through the origin in d + 1 dimensions. When no such hyperplane exists on the sample the data is not linearly separable — and a sign rule has nothing to return.

Learning only from mistakes turns minimisation into feasibility on separable data, and each update buys that point y_i w^T x_i → y_i w^T x_i + ‖x_i‖². Novikoff's bound caps the mistakes at (R/γ)² — dimension-free, priced in geometry — and on non-separable data the rule cycles forever.

The logistic model asserts the log-odds log(p/(1 − p)) are affine in the features, and the sigmoid is the logit's inverse — a smooth, strictly increasing squash where predicting p = w^T x + b outright would sometimes promise 1.3 or −0.2. The old sign boundary z = 0 is exactly even odds.

Maximum likelihood supplies the loss, and its gradient X^T(p̂ − y) is a residual — the same shape least squares carried. The 0–1 loss cannot see this fit at all, and the truth is the loss's own favourite: reporting the actual frequency minimises it.

w_{t+1} = (X^T S_t X)^{-1}X^T S_t z_t with S_t = diag(p̂_i(1 − p̂_i)) and working response z_t — each iterate regresses a locally linearised target. Fisher scoring is the same iteration read statistically, and on separable data there is no minimiser: the weights run to infinity unless a quadratic penalty restores one.

Choose the family (Bernoulli, Poisson, Gaussian) and the link; the canonical link wires the natural parameter itself linearly and leaves the score equation a bare residual X^T(μ̂ − y) = 0 — the same certificate OLS carried. One IRLS fitter serves the whole table; offsets, dispersion and the link choice are the freedom it does not spend for you.

The discriminant δ_c(x) = x^TΣ^{-1}μ_c − ½μ_c^TΣ^{-1}μ_c + log π_c is affine in x, and the posterior is a logistic of the score difference — logistic regression's functional form arrived at from the generative side, fitted by the plug-in principle: closed-form counts and averages, no iteration. Give each class its own Σ_c and the discriminant gains −½(x − μ_c)^TΣ_c^{-1}(x − μ_c) — QDA's parabola.

Ng–Jordan's comparison: the discriminative model's asymptotic risk is never worse, but it approaches it like √(d/n) where the generative reaches its (higher) floor in samples logarithmic in d — scarce data rewards the stronger assumption. The worked two-synonym text dataset shows the curves crossing exactly as the rates predict.

The covariance is what does not scale, so the assumption replaces it with per-class marginals — false almost always, and the product still ranks right often. The probabilities are overconfident even when the ranking is right (repeated evidence multiplies), so read the output as a score, not a probability; smoothing — θ̂_jc = (count + α)/(N_c + α|V|) — keeps an unseen word from vetoing a class.

z = Wx carries the logits and the loss is lse(z) − z_y — feed it logits, not probabilities, because raw scores overflow. Only differences are fitted: shifting every logit by a constant changes nothing, so pin a reference class or keep all K rows and regularise. One-vs-rest answers a different question — K independent fits, no shared normaliser — and is the right shape for multi-label data.

Nonparametric — nothing is fitted, the sample is the model and prediction pays the whole bill at query time. 'Near' is a norm's opinion: the same records re-measured in different units get different neighbours and a different answer, and a Voronoi cell is what one point's neighbourhood looks like.

The strongest guarantee any simple classifier owns — the nearest neighbour's label is a decent proxy for the point's own, when neighbours are near. In dimension d the premise fails, which is why the bound is asymptotic and the curse is next door.

Three effects compound — a cube holding a fixed share of the mass spans nearly every feature's whole range, the nearest of n points sits near the boundary, and distances concentrate until the farthest is barely farther than the nearest. What survives is low intrinsic dimension — the exponent that matters is the data's own — and models that are not local.

Every headline metric is a ratio over these four counts, and each rate conditions on a different half of the table: TPR and FPR on the truth's side (stable under prevalence), precision and NPV on the prediction's side (carrying it). Undefined is a real score — a rule that flags nothing has no precision.

The fraud-screen worked example inverts two rules' verdict depending on which rate is read — and precision is a posterior, so it carries the prior inside where recall does not. F_β prices the trade as a weighted harmonic mean, and macro-F_1 averages per-class F_1's rather than F_1 of the averaged counts.

Both axes condition on the truth, so prevalence never enters — invariant to the class mix, and blind to what a flag actually costs. Crossing curves mean the better ranker depends on where on the axis you stand, and the convex hull marks which points are optimal under some cost ratio.

One sorted pass over the audit set computes both the staircase and its area. Equal AUC does not mean equal ranking — two curves can share the area and trade places region by region — and the number says nothing about thresholds or costs.

A 1-in-1,000 screener's precision collapses at useful recalls — the moving baseline is the feature, not a defect, showing what ROC's truth-conditioned axes hide exactly when positives are rare. Average precision compresses it to Σ_k (R_k − R_{k−1})P_k, and between two measured points precision is not a straight line — no interpolation.

Strictly proper makes the truth the unique minimiser: log loss pays the cross-entropy of the report, the Brier score completes the same square. Accuracy and the linear score reward exaggeration, so a thresholded metric never sees the overconfidence a proper score prices.

The reliability diagram plots acc(b) against conf(b) per bin and ECE = Σ_b(|b|/n)|acc − conf| prices the gaps; the held-out Brier score splits into calibration plus refinement. Measure and repair on data the model never saw.

Every threshold is a claim about costs — the p ≥ ½ default silently prices a false alarm equal to a miss. On the screening table the ratio lands inside a gap in the score's range, and the default can be worse than reviewing everyone; with a headcount instead of costs, the review budget is the threshold.

Deterministic oversampling applied to the loss rather than the data: same ledger, no synthetic rows. It teaches the model a different prior, so the reported probabilities inherit the artificial mix unless rescaled — and SMOTE's interpolated minorities assume the segment between two rare points is safe to cross.

The functional margin scales with ‖w‖, the geometric one is invariant — one spare degree of freedom is spent fixing min_i γ̂_i = 1 (the canonical scale), and then maximising the worst-case margin is the quadratic programme min ½‖w‖². The margin hyperplanes are where y_i(w^T x_i + b) = ±1.

w* = Σ_i α_i y_i x_i is rebuilt from them alone; complementary slackness zeroes every other multiplier, so solving the dual is a support-set guess and KKT sorts the soft-margin points into free, bounded and interior. Six points in the worked example, and only three of them matter.

y_i(w^T x_i + b) ≥ 1 − ξ_i lets a point sit inside the margin or past the boundary — ξ_i > 1 is a misclassification under the margin's units. The primal charges C·Σξ_i, the dual merely caps α_i ≤ C, and eliminating the debts reveals the hinge loss that was being minimised all along.

Minimising ½‖w‖² + CΣξ_i over the slacks is exactly minimising Σ(1 − y_i f_i)_+ — margin debt and surrogate loss are one machine read two ways. Its flat region for m ≥ 1 is what buys the sparse dual.

On the margin axis each stand-in buys and bills differently: hinge buys sparsity and sells probabilities, logistic buys probabilities and never stops paying, exponential buys smoothness and sells robustness, squared is the only non-monotone entry. Classification-calibrated is the property that the surrogate's minimiser returns the right sign — the four floors over the three-band table show the surcharge each pays.

The polynomial kernel (x^T z + c)^p prices p-way interactions in one evaluation; the Gaussian e^{−γ‖x−z‖²} corresponds to an infinite-dimensional φ. The kernel matrix K_ij = K(x_i, x_j) is the quadratic-sized table the method actually pays — fit keeps it, prediction consults it.

The test admits kernels without naming φ — and the algebra helps build them: sums, positive scaling, products, g(x)g(z) factors and pointwise limits all preserve it. The older 'Mercer kernel' class is narrower than the PSD kernels used here; for the kernels of this course the tests agree.

Split w into the span of the training images plus a perpendicular part: the loss's inner products cannot see the perpendicular part and the penalty only grows from it, so it must be zero. The objective reduces to functions of the kernel matrix — kernel ridge regression is the theorem's closed form, paying in n rather than the feature dimension.

The guarantee is about the sample, not the world — n ≥ (ln|ℋ| + ln(1/δ))/ε rows suffice for a realizable finite class, because a bad rule must look perfect on every draw. The bound is a certificate, not the truth: the worked stumps over a four-point grid show both the union bound and what it buys.

Thresholds: 1. Intervals: 2. Half-planes: 3. Axis-aligned rectangles: 4. Hyperplanes in ℝ^d: d + 1 — a property of the class, not its parametrisation. One shattered set gives the lower bound; the no-labelling obstruction gives the upper.

Infinity overcounts — what matters is how many binary patterns the class produces on n points. Sauer–Shelah bounds it polynomially past the VC dimension d: Π_ℋ(n) ≤ Σ_{i ≤ d} C(n, i), which is what lets d_VC replace ln|ℋ| in the bound.

Splitting δ across a nested family by a weight function buys one simultaneous bound — the weight schedule is a prior over classes — and bound by bound the choice has a floor, producing the classical U. A norm penalty is a constraint read sideways, and a margin is capacity measured in geometry.

On any unseen input, half of all labellings refute the prediction; the finite-sample form says some realizable task gives every learner R ≥ 1/8 with probability ≥ 1/7 — the four-point arena counts all sixteen labellings. The theorem buys humility, not nihilism: inductive bias is the only source of advantage.

Gini prices a random guess at the node's own odds, entropy gain is the mutual information between question and label, and misclassification error cannot see purity — a split that only rearranges errors scores zero. The tree predicts the constant of the rectangle you land in: f̂(x) = Σ_m ĉ_m 1[x ∈ R_m] over axis-aligned cells.

Early stopping cannot see past the next split, so the tree is grown deep and pruned: the weakest link is the subtree whose collapse costs least per leaf, its break-even α computable exactly, and the nested collapse order is fixed before α is chosen — which is itself a model-selection problem.

Independent resamples would drive the variance to zero; the shared sample correlates the trees, so the average floors at ρσ². Each resample leaves about e^{−1} ≈ 36.8% of rows unseen — the out-of-bag set that grades the ensemble without a holdout, slightly pessimistically since each voter saw less data.

All the trees' correlation rides through the shared sample; forcing every split to ignore most columns decorrelates the votes, so a forest's variance falls further than bagging's floor. The bias cost is real — a given feature sits a draw out with probability (d − m)/d — and impurity-based importance counts in-sample credit, which is not the same as importance.

The exponential loss e^{−yF} turns focus into weight — each round re-weights rows by their current margin, so the training error falls exponentially in rounds on separable data. A stump suffices as the weak learner; the recipe is tied to its loss, which hears label noise as insistence.

The loss's gradient is a vector over the n rows, but the model must be a function — so each stage fits a tree to the pseudo-residuals by least squares and prices its leaves γ_jm exactly. Shrinkage ν < 1 scales each stage's contribution, trading steps for stability.

Then the leaf weight is closed form, w_j* = −G_j/(H_j + λ), and a split's gain is the children-minus-parents score — the fit reduces to accumulated gradient sums, which is what histogram binning, leaf-wise growth, one-side sampling and exclusive feature bundling make fast.

In-sample predictions leak — a learner's score on its own training rows flatters it — so stacking's level-1 design matrix is built cross-validated. The combiner is a small constrained regression over Z; blending, a single holdout instead of k folds, is the cheaper cousin.

A split threshold reads rank order only — take logs, square positives, rescale: nothing moves. A tabular column carries no geometry for a distance to measure, which is why trees hold the tabular benchmark: irrelevant features are filtered, not averaged, and mixed types need no common scale.

Both half-steps decrease J, so the iteration terminates — at a local minimum the six-point grid demonstrates. Nearest-centre regions are Voronoi cells; k-means++ seeds centres far apart with probability ∝ D(x)², skipping the starts that land badly.

a is the mean distance to its own cluster, b to the nearest foreign one; the score averages s(i) over all n. The silhouette prefers blobs — a ring around a disk reads poorly — and the elbow heuristic it complements reads the bend in an always-falling SSE curve.

Core points anchor DBSCAN's clusters, border points sit in a core's neighbourhood without owning enough, and noise owns neither — density-reachability chains the rest, so the core partition is order-invariant while contested border points go to whichever cluster arrives first. The k-distance plot's knee chooses ε; one radius cannot serve two densities.

π_k N(x_i; μ_k, Σ_k) renormalised over the components — softness lives at the boundary, and the effective membership N_k = Σ_i r_ik the M-step divides by need not match the mixing weights. A component pinning a single point drives its variance to zero and the likelihood to infinity: the collapse singularity the fit bounds rather than solves.

Each step is an exact maximisation that cannot decrease the likelihood — the guarantee buys ascent, not a good answer: the climb is monotone and the summit local, and initialisation is part of the model. At zero variance the responsibilities harden to assignments and the iteration is exactly k-means.

log p(x; θ) = ℒ(q, θ) + KL(q ‖ p(z | x; θ)) — the gap is exactly the KL to the true posterior, so the E-step sets q to that posterior and the M-step maximises the expected complete log Q(θ; θ_old). A decrease after either step is a bug in the code, not the theorem.

One SVD of the centred matrix answers both questions: S = X_c^T X_c/n = V(D²/n)V^T, with direction v_j carrying λ_j = d_j²/n of projected variance. The explained-variance ratio ρ_j reads the scree; the Kaiser rule keeps components above the correlation matrix's average. The answer is a subspace, not a signed arrow — and whitening (decorrelate, then equalise) is the same directions made spherical.

Each point's σ_i is calibrated so its neighbourhood hits a fixed perplexity — the effective neighbour count 2^{H(p_i)} — and the heavy tail buys room in the plane against the crowding problem. Cluster sizes are normalised away and between-cluster distances are free variables: the plot is for hypothesis generation, not evidence.

Only the observed entries Ω enter the sums — a ratings matrix is a table of holes — and each half-sweep is an ordinary regularised least squares. Implicit feedback prices presence against absence with a confidence c_ij = 1 + αr_ij: a missing play is weak evidence, not a zero rating.

c(n) = 2H(n − 1) − 2(n − 1)/n is a random tree's mean external depth, making 0.5 unremarkable and 1 immediate. Distance methods mask (a dense anomaly cluster hides its members) and swamp (normals near the crowd look anomalous); isolation replaces distance with depth — the one-day-ten-meters trace scores the spike 0.792.

Single linkage chains — it merges at the closest pair and can string scattered points together; complete prices the worst pair; average sits between; Ward pays Δ = n_G n_H/(n_G + n_H)·‖μ_G − μ_H‖², the SSE the merge adds. The dendrogram remembers every merge's height, and the Lance–Williams update re-prices the matrix without rescanning it.

Prior w ~ N(0, τ²I) plus Gaussian likelihood completes the square; the posterior mean μ_N is exactly the ridge solution with λ = σ²/τ² — the penalty was a prior all along. The predictive law adds x_*^T Σ_N x_* to the noise floor: an interval that widens away from the data.

The log evidence splits into a best-fit term minus a complexity term — the Occam penalty lives there, because a flexible model spreads its prior thin. Log-evidence differences are log Bayes factors, empirical Bayes fits the prior's knobs to the evidence itself, and the worked degree sweep shows evidence choosing where fit alone cannot.

Conditioning gives the predictive law μ_* = k_*^T K_y^{-1}y, σ_*² = k_{**} − k_*^T K_y^{-1}k_* — the band is prior belief, not measured error. The length-scale ℓ decides how far correlation reaches and the evidence picks it; exactness bills as n³.

Expected improvement EI(x) = (μ(x) − f⁺)Φ(z) + σ(x)φ(z) prices promising means and uncertain ones together — the loop spends computation instead of trials, and still loses to random search when the surrogate's prior is wrong.

Missing edges are what the graph saves — the full table is exponential in the variables the factorisation avoids. Fitting the conditional probability tables is counting within rows; ancestral sampling runs the net as a machine for synthetic days; inference sums the hidden variables out. Arrows describe, they do not prove.

Seeing the middle blocks a chain or a fork; a collider is silent until you look at it — then conditioning opens it, the cold explaining away the allergy, and a conditioned descendant opens it just the same. d-separation checks every node on every path: blocked when a non-collider is conditioned, or a collider is unconditioned with no conditioned descendant.

Scoring a sequence naively means summing K^T hidden paths — the forward pass prices each prefix once and reuses it: α_t(j) = p(x_1..x_t, z_t = j), computed in K²T. Long products underflow, so every column is rescaled on the way through.

V_t(j) = b_j(x_t)·max_i[V_{t−1}(i)A_ij] — the forward pass with a max and a memory, ψ_t(j) keeping the winning predecessor for the backtrace. The per-position mode can name an impossible path: decoding is a joint question.

γ_t(i) = p(z_t = i | x) prices the cell, ξ_t(i, j) = p(z_t = i, z_{t+1} = j | x) the edge — the backward pass is what makes them cheap — and the M-step re-estimates π, A and b with fractional counts. The climb is monotone and the summit is local: initialisation is part of the model.

A July model repriced on December rows — the risk certificates were all computed inside one distribution, and the factorisations p(y|x)p(x) versus p(x|y)p(y) name which piece moved. Detection is a two-sample problem over unlabelled streams: a domain classifier's AUC_dom is the statistic that can see it.

Only the class prior drifts, so the correction reweights the posterior at the new mix — provided p(x|y) truly held. Counting predictions is not counting labels, but predictions carry the prior through a known linear map, which is the handle the correction pulls.

The one shift no input statistic can see, since x's law never changed; only the outcome errors over time carry the signal. Monitoring can promise detection of input drift and cannot promise it for meaning drift.

Reweighting retargets the average: E_train[w·ℓ] = E_test[ℓ] under support inclusion — a test region the training never saw cannot be reweighted in. The ratio is a domain posterior up to constants, and the worked two-rule example flips their ranking between training and deployment pricing.

Weights are paid for in effective samples — a few huge ratios leave the estimate carried by a handful of rows. Clipping buys rows back at the price of bias, and flattening the ratio is the dial between the two.

The prediction distribution is the confusion matrix times the deployed prior — counting predictions and solving the linear system recovers q, and the two-class case collapses to a single subtraction. Where the confusion matrix is near-singular the inversion misleads; the EM variant uses posteriors instead of decisions.

The marginal slope of y on x_1 is w_1 + w_2·Cov(x_1, x_2)/Var(x_1): with correlated columns the coefficient prices only the orthogonal remainder, so correlated features flip signs and VIF_j = 1/(1 − R_j²) prices the inflation. Magnitude as importance, selection as discovery, and any causal claim all outrun what the glass box licenses.

The shuffled rows become impossible inputs and the model reads the copy — that is the measurement. Correlated columns break the shuffle twice, understating a redundant twin, and the repeat variance over π is part of the answer, not noise.

Efficiency, symmetry, dummy and additivity pin down this one split — the proof builds the game space from unanimity games. 'Not knowing a feature' is a modelling choice: interventional and conditional shuffles answer different questions, and TreeSHAP beats the 2^d sum only by exploiting the model's shape.

The decision ignores the protected attribute entirely; on the lesson's shared-threshold scorecard it fails even for a calibrated score. Fairness through unawareness is not a route — the group coordinate leaks through correlates whether or not the model reads it.

Equalised odds in full; equal opportunity is the positive half (TPR parity). On the shared-threshold scorecard separation fails asymmetrically — the errors each group pays are different kinds — and a per-group confusion matrix still cannot see inside a score's construction.

For a score R rather than a decision the demand is calibration by group: P(Y = 1 | R = r, A = a) = r. Sufficiency is about the map's meaning, not its provenance — the score can be perfectly calibrated per group and still collide with the error-rate criteria.

One line of conditional probability applied twice — bookkeeping, not ethics — and it forces the impossibility: equal PPV and equal error rates across two groups coexist only at equal base rates or a perfect predictor. The COMPAS debate ran exactly along the fault line: the FPR gap and the calibration were both accurately measured.

Schema, baseline, features, audit, comparison, calibration, threshold, monitoring, explanation: each row names the decision and the numbers that justified it, so a retrain or an audit checks the reasoning, not just the artifact.

t* = c/(ρV): offer cost over saveable margin. A constraint on how many flags the team can work changes the selection rule (top-k by score), not the price.

The battery: a learning curve's plateau bounds what more data earns, sliced errors locate the mass, a label audit questions the ground truth itself, leakage and shift checks certify the meter, and paired ablations price each component. The four-way spend — data, features, capacity, metric — is answered from priced evidence, not preference.

Applied to the label feed in the case study: sampling both disagreement and agreement cells recovered a third of the observed miss — the bug was in the meter, not the model.

Ask first whether labels exist at all — without them the task is unsupervised (clustering or density estimation) — then what one answer looks like. The shape of Y, not the algorithm, names the task.

Three tariff plans become three columns with a single 1 — every level equidistant. An integer code asserts spacing the data never gave; with a constant term, keep k−1 columns and read the baseline off the intercept.

Band j of m lights its first j columns, so adjacent levels differ in one coordinate — 'warm is at least mild' without a numeric scale. An integer code would claim unit spacing the bands do not have.

Weekday d becomes (sin 2πd/7, cos 2πd/7): every day sits the same distance, 2 sin(π/7) ≈ 0.87, from the next. An integer code puts Monday and Sunday maximally far apart.

Temperature × weekend lets the same reading mean different usage on a Saturday. Keep the main-effect columns too, so the product modulates rather than replaces them.

The choice decides which summary is optimal — squared risk is minimised by the conditional mean, absolute by the median — and a few large misses dominate the square where the absolute shrugs them off.

Every wrong label costs 1, near-miss or not — the honest score for classification, but a flat shape descent methods cannot follow; enumerable classes still minimise it directly, which is what ERM does, and the surrogate question is forwarded for the rest.

On the three-band table the floors are 243.375 squared, 11.25 absolute and 0.425 under 0–1 — the earlier table's band lookup pays 0.55, already behind the best possible before any fitting is discussed.

Alarm when the miss's cost-weighted chance beats the false alarm's: at 5:1 the boundary is p > 1/6, and the symmetric rule pays 1.015 where the costed rule pays 0.465.

The training score is biased low by construction; a held-out mean of m losses is unbiased for that rule's risk. Consulting the same set for decisions promotes it to a validation set — and the bias returns.

At an 8% error rate, ±0.01 needs 2828 rows and ±0.005 needs 11310 — half the width costs four times the rows. Planning blind, p̂ = 1/2 gives the conservative budget.

For equal folds it prices the procedure at (k−1)n/k rows; pessimism relative to full-size training depends on a decreasing learning curve. Shuffle interchangeable rows, stratify scarce levels, and group dependent rows. The naive fold-variance estimate is biased down with positive covariance and up with negative covariance; dependent refits do not automatically yield a valid error bar.

The normal equations give the exact minimiser in one solve; at large d, or where an iterative fit is anyway the machinery, gradient descent replaces it — and then the features' scales set the step count through the Hessian's condition number.

OLS's predictions do not change under rescaling the columns; a norm penalty's price and a distance's notion of near both do — metres against millimetres multiplies the condition number by 10⁶. Fit the scaler on the training rows and carry it inside the fold.

The solver is unchanged — ŷ = Σw_jb_j(x) is still linear in w — but p + 1 = n interpolates the noise, so pair the expansion with a penalty or a validation choice of degree.

The decomposition σ² + bias² + Var splits the expected error: high training error is the bias signature and buys nothing from extra rows; a train–test gap is variance, which data does buy down.

The quadratic penalty is rotation-friendly, never zeroes a weight, and fades effective degrees of freedom smoothly; the ℓ₁ ball's corners put weights at exactly zero — selection inside the fit — but among correlated twins it picks arbitrarily, where the elastic net shares the credit.

Reserve a test set touched once, or score the whole choosing procedure by nested CV; among candidates within one standard error of the best, take the simplest — overlapping CV error bars mean the edge is often noise.

d_i = ℓ_A(i) − ℓ_B(i) cancels the test set's luck; for 0–1 losses McNemar counts only the discordant pairs; a t-test over fold scores overstates confidence because the training sets overlap.

A grid re-samples the useless axis; random covers all axes per trial — T ≥ ln(1 − α)/ln(1 − q) trials buy a top-q hit at confidence α. Halving wins when curves separate early; it lies for slow starters that cross late.

The four leak channels — preprocessing fitted outside, features from the future, label-derived columns, repeated entities across the split — are all blocked by the same rule: the pipeline is one composite object, fitted on the fold's training share only.

Whatever the family, run the screen inside the CV loop — selecting on all rows and then validating reports pure noise as predictive (0.548 against 0.048 in the lesson's simulation).

The perceptron and the SVM return verdicts; the logistic fit prices confidence — the price is that its log loss keeps charging overconfidence forever. And always fit it penalised: on separable data the unpenalised weights run to infinity.

The generative model reaches its asymptote in roughly log(d) samples where the discriminative approaches its better floor like √(d/n) — the worked two-synonym dataset shows the curves crossing; plentiful data forgives the weaker assumption.

Conditional independence is false almost always, yet the ranking survives; the probabilities are overconfident because repeated evidence multiplies — smooth the tables (α pseudo-counts), and calibrate before the scores mean anything.

Softmax's logits share a normaliser, so the outputs form a distribution over a partition; one-vs-rest trains K independent questions and suits multi-label data where labels are not exclusive.

Nothing is trained — the sample is the model, and Cover–Hart guarantees asymptotic risk within twice the Bayes floor. Standardise first: 'near' is a norm's opinion, and in high dimension the nearest point stops being near.

Truth-conditioned rates ignore prevalence; prediction-conditioned ones carry it — the fraud screen's two rules swap verdicts depending on which is read, and F_β exists to price the trade explicitly.

AUC = P(s_+ > s_-) never sees prevalence — a feature for comparing rankers, a blind spot for pricing a deployed alert. The PR curve's moving baseline is the information: it shows what ROC hides exactly when a 1-in-1,000 screener is the question.

Only proper rules make honesty the unique minimiser; accuracy and the linear score reward exaggeration. The Brier decomposition splits what you pay for miscalibration from what the forecast's resolution earns back — climatology is the anchor to beat.

Both are one more fit, on a calibration set the model never saw — Platt spends two parameters, isotonic spends monotonicity alone and can conform to any repair shape.

Both teach the model a different prior, so rescale the reported probabilities afterwards or the scores inherit the artificial mix; SMOTE's interpolated minorities further assume the segment between two rare points is safe to cross.

The dual sees the data only as inner products and prices n unknowns against the primal's d + 1, flipping the cost once a feature map inflates d; α_i > 0 names the support vectors the boundary depends on.

Each sells something too — hinge sells probabilities, logistic never stops paying, exponential sells robustness to label noise, squared alone is non-monotone. Classification calibration is the guarantee that the minimiser returns the right sign.

Polynomial kernels price finite interaction order in one evaluation; the Gaussian's feature map is infinite-dimensional. Any proposed kernel must induce positive-semidefinite Gram matrices — the Mercer condition — or it is not one.

Splitting δ across classes by a weight schedule buys one simultaneous guarantee; a norm penalty is the same control read as a Lagrange multiplier, and a margin is capacity measured in geometry.

Averaged over all tasks every learner ties — some realizable task gives any learner R ≥ 1/8 with probability ≥ 1/7 — so an apparent algorithmic edge is a bet on which tasks are real, not a free lunch.

Early stopping cannot see past the next split; cost-complexity R̂(T) + α|T| prices each leaf, and the collapse order is computable exactly before α is chosen — which is itself a model-selection problem.

Bagging's variance falls as ρσ² + (1 − ρ)σ²/B and floors at the shared-sample correlation — the out-of-bag rows are a free validation set. The random forest's per-node lottery of m of d columns breaks that correlation, at a real bias cost: a given feature sits a draw out with probability (d − m)/d.

AdaBoost's exponential loss re-weights by the margin and hears label noise as insistence; gradient boosting takes any differentiable loss via pseudo-residuals, with shrinkage ν trading steps for stability. At scale, the quadratic stage — leaf weights −G/(H + λ), splits scanned over histogram bins, leaf-wise growth — is what makes exact split search affordable.

The combiner must train on out-of-fold predictions — in-sample scores leak into the level-1 matrix; a single holdout (blending) is the cheaper cousin when the refits are too dear.

Lloyd's iteration descends J to a local minimum, so the start matters; the squared distance inherits the metric (standardise first) and the objective never chooses k — that choice stays yours. Where points sit between clusters, a Gaussian mixture's responsibilities price the boundary softly — at zero variance it is k-means — but watch the collapse singularity: a component pinned to one point makes the likelihood unbounded.

SSE always falls, so the elbow reads the bend; the silhouette prefers blobs; bootstrap agreement prices repeatability not correctness; ARI needs a reference partition that is itself a choice.

The ε–minPts rule sorts points into core, border and noise; the k-distance plot's knee sets ε. One radius cannot serve two densities — that is hierarchical density's territory (HDBSCAN).

PCA answers with a subspace and explained-variance accounts; t-SNE normalises cluster sizes away and frees between-cluster distances — a hypothesis generator, not evidence.

A dense anomaly cluster hides its members from density and distance scores (masking) and marks normals near the crowd (swamping); isolation replaces distance with depth — s = 2^{−E[h]/c(n)} — and with almost no labels, judge the ranker by precision@k.

The posterior is closed form and ridge is its mean — the same answer with an honest uncertainty statement attached; at the boundary, where a distribution over functions is wanted, the Gaussian process takes over.

The kernel is the prior — the model is the covariance you assert; the evidence picks the length-scale and noise. The band is prior belief, not measured error, and the cubic solve is the price of no approximation.

Missing edges are the saving — the full joint is exponential in the variables the factorisation avoids; inference sums hidden variables out, exact but priced by the table widths.

Conditioning on a collider or its descendant opens a path that was closed: 'adjust for X' can create the bias it was meant to remove. Enumerate paths; each is blocked by a conditioned non-collider or an unconditioned collider.

The forward pass prices a sequence in K²T, Viterbi decodes the single best path with a max and a memory, and Baum–Welch fits the tables on fractional counts — the climb is monotone, the summit local, so initialisation is part of the model.

The remedy depends on which piece walked — reweight the rows, re-prior the posterior, or retrain outright. A domain classifier's AUC detects input moves; concept drift leaves no trace in the inputs at all.

Support inclusion is load-bearing: a region of the test distribution the training never saw cannot be reweighted in; clipping the ratios buys effective rows back at the price of bias.

Count predictions, not labels — the confusion matrix maps the deployed prior to the deployed prediction mix; the two-class case is one subtraction, and a near-singular C makes the inversion unreliable.

Correlated columns flip signs and inflate variances — VIF prices it — and the fitted structure is one draw of the data, so a tree's path or a weight's rank is a sample, not a mechanism.

Permutation importance understates correlated twins twice over; Shapley's axioms pin one answer but 'not knowing a feature' is a modelling choice — interventional and conditional versions answer different questions.

Fairness through unawareness is not a route — the attribute leaks through correlates — and each criterion can be bought only by trading another; a per-group confusion matrix still cannot see inside a score's construction.

Equal PPV and equal error rates across groups coexist only at equal base rates or a perfect predictor — the COMPAS dispute ran exactly along this fault line. Relaxing one criterion is a choice; the mathematics can only price it.

Predict all-stay and accuracy looks superb while the value is zero. Price flags instead: caught positives times saveable margin minus flags times offer cost, and flag only above c/(ρV).

The case study's observed gap decomposed into label-feed bugs, a leaked split and a shifted mix; the model was fine. A bigger model bought against a corrupted plateau wastes the budget on the wrong component.