Probability & Statistics

Formulas

Probability theory acquired a rigorous mathematical foundation in 1933, when Andrey Nikolaevich Kolmogorov published "Foundations of the Theory of Probability", laying down the axiomatic basis that remains in effect to this day.

Definition: The triple (Ω, F, P), where: Ω is the sample space of elementary outcomes. F ⊆ 2^Ω is a σ-algebra of events. P: F → [0,1] is a probability measure.

σ-algebra F: A family of subsets of Ω satisfying: (1) Ω ∈ F; (2) A ∈ F ⇒ Aᶜ ∈ F (closure under complements); (3) A₁, A₂,... ∈ F ⇒ ⋃ₙ Aₙ ∈ F (closure under countable unions).

P1 (Non-negativity): P(A) ≥ 0 for all A ∈ F. P2 (Normalization): P(Ω) = 1. P3 (Countable additivity): For pairwise disjoint A₁, A₂,...: P(⋃ₙ Aₙ) = Σₙ P(Aₙ).

Definitions

Event tree

— a visual way to apply conditional probability formulas. Every node is an event, each branch is a conditional transition probability. The probability of a final outcome is the product of probabilities along a path from the root. The sum of the prob...

Formulas

Conditional probability is the probability of an event given that another event has occurred. This allows us to update our knowledge as information arrives and is the foundation of Bayesian inference.

Multiplication Theorem: P(A₁∩A₂∩...∩Aₙ) = P(A₁)·P(A₂|A₁)·P(A₃|A₁A₂)·...·P(Aₙ|A₁...Aₙ₋₁).

Law of Total Probability: If B₁,...,Bₙ form a complete group (mutually exclusive, ⋃Bᵢ = Ω): P(A) = Σᵢ P(A|Bᵢ)P(Bᵢ).

Bayes’ Theorem: P(Bᵢ|A) = P(A|Bᵢ)P(Bᵢ) / Σⱼ P(A|Bⱼ)P(Bⱼ). “Posterior probability of hypothesis Bᵢ upon observing A.”

Formulas

Classical probability assumes a finite number of equally probable outcomes. For continuous spaces (interval, circle, ℝⁿ), geometric probabilities are needed, defined via the Lebesgue measure.

Principle: In a continuous space, “equally possible” means a uniform distribution according to the measure (length, area, volume). P(A) = measure(A)/measure(Ω).

Bertrand’s Paradox (1889): “Random chord” of a circle — is it longer than the side of an inscribed equilateral triangle? Three reasonable definitions of “random chord” yield P = 1/2, P = 1/3, P = 1/4. Demonstrates that “equally possible” is ambiguous without precise definition.

Buffon’s Needle: A needle of length l is thrown onto a floor with parallel lines spaced d apart (l < d). P(crossing a line) = 2l/(πd). Allows estimation of π via experiment!

Formulas

A discrete random variable takes on a countable number of values. It is fully described by its probability mass function (PMF)—the probabilities of each value.

Binomial: X ~ Bin(n,p). X = number of successes in n independent Bernoulli trials. P(X=k) = C(n,k)p^k(1-p)^{n-k}. E[X]=np, Var[X]=np(1-p). Generating function: G(z) = (1-p+pz)^n.

Poisson: X ~ Poisson(λ). P(X=k) = e^{-λ}λ^k/k!, k=0,1,2,... E[X] = Var[X] = λ. Limit of Bin(n,p) as n→∞, p→0, np→λ.

Geometric: X ~ Geom(p). X = number of trials until first success. P(X=k) = (1-p)^{k-1}p. E[X]=1/p, Var[X]=(1-p)/p². Memoryless property: P(X>m+n|X>m) = P(X>n).

Definitions

Discrete analogue

— geometric distribution (P(X > m+n | X > m) = P(X > n)). This explains why Poisson processes (continuous time, exponential inter-event intervals) and Markov chains (discrete time, geometric holding times) are mathematically close.

Formulas

Continuous random variable has a probability density function f(x) ≥ 0 with ∫f dx = 1. Probability of falling into an interval is the integral of the density.

X ~ N(μ, σ²): f(x) = (1/√(2πσ²)) exp(-(x-μ)²/(2σ²)). E[X]=μ, Var[X]=σ². Symmetric, 68-95-99.7 rule (±1σ, ±2σ, ±3σ).

Standard: Z ~ N(0,1). Function Φ(z) = P(Z≤z). P(a<X<b) = Φ((b-μ)/σ) - Φ((a-μ)/σ).

Sum of normals: If X~N(μ₁,σ₁²) and Y~N(μ₂,σ₂²) are independent: X+Y~N(μ₁+μ₂, σ₁²+σ₂²).

Formulas

Multivariate distributions describe the joint behavior of several random variables. Copulas are a powerful tool for modeling dependencies between variables independently of their marginal distributions.

Joint Density: (X, Y) ~ N₂(μ, Σ). Σ = [[σ₁², ρσ₁σ₂], [ρσ₁σ₂, σ₂²]]. The correlation parameter ρ = Cov(X, Y)/(σ₁σ₂). Conditionals: Y|X=x ~ N(μ₂ + ρσ₂/σ₁(x−μ₁), σ₂²(1−ρ²)).

Correlation ≠ dependence: For ρ=0: X and Y are uncorrelated, but not necessarily independent (for normals — this is equivalent!).

Sklar’s Theorem (1959): For the joint CDF H(x, y) = C(F₁(x), F₂(y)), where F₁, F₂ are marginal CDFs, C: [0,1]² → [0,1] is a copula. With continuous marginals, C is unique.

Formulas

Mathematical expectation is the "center of mass" of a distribution. Moments characterize the shape of a distribution: mean, variance, skewness, kurtosis.

Definition: $E[X] = \sum x \cdot P(X=x)$ (discrete), $\int x \cdot f(x)dx$ (continuous). Exists if $\sum|x|P(X=x) < \infty$.

Linearity: $E[aX+bY] = aE[X] + bE[Y]$ — always (regardless of the dependence of $X, Y$!). $E[X_1+\dots+X_n] = n\mu$ (if $E[X_i]=\mu$ for all $i$).

For functions: $E[g(X)] = \sum g(x)P(X=x) = \int g(x)f(x)dx$. Jensen’s inequality: for convex $g$: $E[g(X)] \geq g(E[X])$.

Formulas

Generating functions are a powerful tool for working with distributions of the sum of independent random variables. The characteristic function always exists and uniquely determines the distribution.

Definition: $M_X(t) = E[e^{tX}] = \sum E[X^n] t^n / n!$ (Taylor series). If it exists in a neighborhood of $t=0$: $E[X^n] = M_X^{(n)}(0)$. The MGF uniquely determines the distribution.

For the sum of independents: $M_{X+Y}(t) = M_X(t)\cdot M_Y(t)$. This turns convolutions into multiplications.

Examples: Poisson$(\lambda)$: $M(t) = \exp(\lambda(e^t-1))$. $N(\mu,\sigma^2)$: $M(t) = \exp(\mu t + \sigma^2 t^2 / 2)$. Sum of normal distributions: $M_{X+Y} = \exp((\mu_1+\mu_2)t + (\sigma_1^2+\sigma_2^2)t^2/2) \to N(\mu_1+\mu_2, \sigma_1^2+\sigma_2^2)$. ✓

Formulas

Probability inequalities allow us to estimate event probabilities without full knowledge of the distribution. They are critically important for statistics, machine learning, and information theory.

Markov's Inequality: For $X \geq 0$, $a > 0$: $P(X \geq a) \leq \mathbb{E}[X]/a$. Proof: $\mathbb{E}[X] \geq \mathbb{E}[X \cdot 1_{X \geq a}] \geq a \cdot P(X \geq a)$.

Chebyshev's Inequality: For any $X$ with finite $\mathbb{E}[X] = \mu$, $\operatorname{Var}[X] = \sigma^2$: $P(|X-\mu| \geq k) \leq \sigma^2/k^2$ or $P(|X-\mu| \geq k\sigma) \leq 1/k^2$. Does not require knowledge of the distribution, only $\mu$ and $\sigma^2$.

Jensen's Inequality: For convex $g$: $\mathbb{E}[g(X)] \geq g(\mathbb{E}[X])$. Corollaries: $\mathbb{E}[X^2] \geq (\mathbb{E}[X])^2$ (variance $\geq 0$). $\mathbb{E}[e^X] \geq e^{\mathbb{E}[X]}$.

Formulas

The Law of Large Numbers (LLN) is a fundamental result in probability theory: the mean of a large number of independent random variables converges to their expected value. This is the mathematical foundation of statistics.

Weak LLN (Chebyshev, 1866): For i.i.d. $X_1,\ldots,X_n$ with $E[X_i]=\mu$ and $Var[X_i]=\sigma^2 < \infty$: $\overline{X}_n = (X_1+\ldots+X_n)/n \to_P \mu$ (convergence in probability). Proof via Chebyshev: $P(|\overline{X}_n-\mu|\geq\epsilon) \leq \sigma^2/(n\epsilon^2)\to 0$.

Strong LLN (Kolmogorov): Under the same conditions: $\overline{X}_n \to \mu$ almost surely ($P(\lim \overline{X}_n = \mu) = 1$). This is stronger than the weak law (almost sure $\to$ in probability, but not vice versa).

LLN without finite variance (Khinchin): Finiteness of $E[|X|]$ is sufficient. For $Var[X]=\infty$, LLN still holds!

Formulas

Central Limit Theorem (CLT) is one of the most important results in mathematics. It explains why the normal distribution is ubiquitous: the sum of a large number of independent random variables approaches the normal distribution.

Theorem (Lindeberg-Levy): For i.i.d. $X_1,\ldots,X_n$ with $E[X_i]=\mu$, $Var[X_i]=\sigma^2 < \infty$: $(\bar{X}_n - \mu)/(\sigma/\sqrt{n}) \xrightarrow{d} N(0,1)$. Equivalently: $(S_n - n\mu)/(\sigma\sqrt{n}) \xrightarrow{d} N(0,1)$.

Rate of convergence (Berry-Esseen): $|P((S_n-n\mu)/(\sigma\sqrt{n}) \leq x) - \Phi(x)| \leq C\cdot\rho/(\sigma^3\sqrt{n})$, where $\rho = E[|X-\mu|^3]$. With $C \leq 0.4748$. Rate $O(1/\sqrt{n})$.

Proof via characteristic functions: $\varphi_{(S_n-n\mu)/(\sigma\sqrt{n})}(t) = [\varphi_X(t/(\sigma\sqrt{n}))]^n$. Expanding $\varphi_X(t) \approx 1 - t^2/2 + \ldots$ for small $t$: $\rightarrow \exp(-t^2/2) = \varphi_{N(0,1)}(t)$. Lévy's continuity theorem $\rightarrow$ CDF converges to normal.

Formulas

There exist several different concepts of convergence for sequences of random variables. Large Deviations Theory studies the probabilities of rare events—exponentially small probabilities of deviations from the mean.

In probability: $X_n \to_P X$ if $\forall \varepsilon > 0:\ P(|X_n-X| > \varepsilon) \to 0$ as $n\to\infty$.

Almost surely (a.s.): $X_n \to_{a.s.} X$ if $P(\lim X_n = X) = 1$. Stronger than in probability. Consequence: convergence in probability.

In distribution (weak): $X_n \to_d X$ if $F_{X_n}(x) \to F_X(x)$ for all points of continuity of $F_X$. The weakest—does not require identity on a single probability space.

Formulas

Statistical estimation is the construction of estimates for unknown parameters based on observed data. The maximum likelihood method, Bayesian approach, and method of moments are the three main paradigms.

Likelihood function: $L(\theta; x_1, ..., x_n) = \prod_i f(x_i; \theta)$ (for an i.i.d. sample). Log-likelihood: $\ell(\theta) = \sum_i \log f(x_i; \theta)$.

MLE: $\hat{\theta}_{\text{MLE}} = \arg\max_\theta \ell(\theta)$. Solution: $\partial\ell/\partial\theta = 0$ (likelihood equations).

Properties of MLE: Consistency: $\hat{\theta}_n \to_P \theta_0$. Asymptotic normality: $\sqrt{n}(\hat{\theta}_n - \theta_0) \to_d N(0, I(\theta_0)^{-1})$. Asymptotic efficiency: attains the Cramer-Rao lower bound.

Formulas

Nonparametric methods do not assume a concrete distributional form. They are used when the assumptions of parametric methods are violated. Bootstrap is a universal technique for error estimation.

Sign Test: H₀: median = m₀. Statistic: number of observations > m₀. Under H₀: ~ Bin(n, 0.5). Does not depend on the distribution form, but is not very efficient.

Wilcoxon Signed-Rank Test: Accounts for the magnitude of deviations. Compute dᵢ = xᵢ − m₀, rank |dᵢ|, W⁺ = sum of ranks with dᵢ > 0. Under H₀: E[W⁺] = n(n+1)/4, Var[W⁺] = n(n+1)(2n+1)/24.

Mann-Whitney Test: Nonparametric two-sample test. U = #{(xᵢ, yⱼ): xᵢ > yⱼ}. Under H₀: E[U] = n₁n₂/2. Analog of two-sample t-test without normality.

Formulas

Sufficient statistics compress all information about a parameter from the sample. The Rao-Blackwell theorem allows improving any estimator by conditionally averaging over a sufficient statistic.

Definition (Fisher, 1922): A statistic T(X) is sufficient for θ if the conditional distribution of the sample given T is independent of θ. Intuition: T contains everything the sample knows about θ.

Factorization Criterion (Neyman-Fisher): T is sufficient if and only if the likelihood factorizes: $L(\theta; x) = g(T(x); \theta) \cdot h(x)$. The part depending on θ enters only via T(x).

Examples: Poisson(λ): $T = \sum X_i$. Bernoulli(p): $T = \sum X_i$. N(μ, known σ²): $T = \bar{X} = \sum X_i / n$. N(μ, unknown σ²): $T = (\sum X_i, \sum X_i^2)$ — a two-dimensional sufficient statistic.

Formulas

Hypothesis testing is a formalized decision-making procedure based on data. The null hypothesis H₀ is rejected or not rejected based on the criterion statistic.

Null (H₀) and alternative (H₁) hypotheses. Significance level α = P(Type I error) = P(reject H₀ | H₀ is true). Power 1-β = P(reject H₀ | H₁ is true).

p-value: p = P(observed statistic or more extreme | H₀). If p < α → reject H₀. p-value ≠ probability of H₀ (misinterpretation!).

Student's t-test: σ unknown. T = (X̄-μ₀)/(S/√n) ~ t(n-1). |T| > t_{α/2,n-1} → reject.

Formulas

Linear regression models the dependence between variables and is one of the most important tools in applied statistics. Analysis of variance generalizes the t-test to multiple groups.

Model: $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$, $\varepsilon_i \sim N(0, \sigma^2)$ i.i.d. OLS estimates: $\hat{\beta}_1 = \Sigma (x_i-\bar{x})(y_i-\bar{y})/\Sigma(x_i-\bar{x})^2$, $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$.

Gauss-Markov Theorem: $\hat{\beta}$ is BLUE (best linear unbiased estimator) among all linear unbiased estimators. $\operatorname{Var}[\hat{\beta}_1] = \sigma^2/\Sigma(x_i-\bar{x})^2$. $S^2 = \operatorname{RSS}/(n-2)$ is an unbiased estimator of $\sigma^2$.

Coefficient of Determination $R^2$: $R^2 = 1 - \operatorname{RSS}/\operatorname{TSS}$, $\operatorname{RSS} = \Sigma(y_i-\hat{y}_i)^2$, $\operatorname{TSS} = \Sigma(y_i-\bar{y})^2$. $R^2 = \operatorname{cor}(y, \hat{y})^2$ in simple regression. The proportion of explained variation $\in [0,1]$.

Formulas

Goodness-of-fit tests assess whether data conform to a hypothesized theoretical distribution. This is an important step in statistical analysis before applying parametric methods.

Idea: Divide the data into k cells with observed frequencies Oᵢ and theoretical Eᵢ = nPᵢ(θ). Statistic: χ² = Σᵢ(Oᵢ - Eᵢ)²/Eᵢ. Under H₀, asymptotically: χ² ~ χ²(k-1-r), where r is the number of estimated parameters.

Conditions for applicability: Eᵢ ≥ 5 for each cell (if violated, combine adjacent cells). Independent observations. Minimum n ≥ 30-50.

Contingency tables: Testing independence of two categorical variables. χ² = Σᵢⱼ(Oᵢⱼ - Eᵢⱼ)²/Eᵢⱼ, Eᵢⱼ = nᵢ·nⱼ/n. Degrees of freedom: (r-1)(c-1).

Definitions

Ergodic chain

— irreducible, aperiodic, positive recurrent. For an ergodic chain: μₙ → π (geometrically fast when S is finite). Mixing rate: spectral gap gap Lₛ = 1 − λ₂ (λ₂ — the second largest eigenvalue of P). Mixing time: τ_mix ≈ 1/gap. For the "shuffling" ch...

Formulas

A Markov chain is a memoryless stochastic process: the future depends only on the present, not on the past. This is a powerful model for systems with "states": queues, markets, genomic sequences, PageRank.

Definition: A sequence X₀, X₁, X₂, ... with values in a finite/countable set S satisfies the Markov property: P(Xₙ₊₁ = j | X₀,...,Xₙ) = P(Xₙ₊₁ = j | Xₙ) for all n and j.

Transition matrix: Pᵢⱼ = P(Xₙ₊₁ = j | Xₙ = i). Stochastic: Pᵢⱼ ≥ 0, ΣⱼPᵢⱼ = 1 for all i.

n-step transitions: P(Xₙ = j | X₀ = i) = (Pⁿ)ᵢⱼ — the (i,j)-th element of the matrix Pⁿ.

Formulas

Poisson process is a standard model of events that occur randomly over time (calls to a call center, atomic decay, transactions). CTMCs describe systems with continuous time and discrete states.

Definition: {N(t), t ≥ 0} is a Poisson process with rate λ if: 1. N(0) = 0 2. Increments on non-overlapping intervals are independent 3. N(t + s) − N(t) ~ Poisson(λs) for all t,s > 0

Inter-event intervals: $T_i$ = time between the i-th and (i-1)-th event ~ Exp(λ), independent.

Superposition: The merging of two Poisson processes with rates λ₁ and λ₂ is a Poisson process with rate λ₁+λ₂. Thinning: each event is independently included with probability p — Poisson with rate pλ.

Formulas

A martingale is a stochastic process without “systematic drift”: the expected future value is equal to the current value. It is the mathematical embodiment of a “fair game” and the foundation of financial mathematics.

Definition: The sequence {Mₙ, Fₙ} is a martingale relative to the filtration {Fₙ} if: (1) Mₙ is Fₙ-measurable; (2) E[|Mₙ|] < ∞; (3) E[Mₙ₊₁|Fₙ] = Mₙ.

Examples: Symmetric random walk: Sₙ = X₁ + ... + Xₙ, Xᵢ = ±1. E[Sₙ₊₁|Fₙ] = Sₙ. Mₙ = Sₙ² - n is also a martingale (suitable for calculating E[τ]).

Optional Stopping Theorem (Doob): Under sufficient conditions for the martingale Mₙ and stopping time τ: E[M_τ] = E[M₀]. “In a fair game the expected gain is zero.”

Formulas

Brownian motion (Wiener process) is a mathematical model of chaotic motion, described by Brown in 1827 while observing pollen on water. It is the limit of random walks and the foundation of stochastic analysis.

Standard Brownian motion: The process {W_t, t ≥ 0} such that: 1. W₀ = 0 2. Independence of increments: W_{t₄}-W_{t₃} ⊥ W_{t₂}-W_{t₁} for 0≤t₁<t₂≤t₃<t₄ 3. Normality: W_t - W_s ~ N(0, t-s) 4. Continuity of trajectories: t → W_t is continuous

Construction (Levy–Centsov): Via Gaussian series over Haar functions: W_t = Σ_{n,k} Z_{nk}·H_{nk}(t). Or via the CLT: W_t = lim_{n→∞} S_{⌊nt⌋}/√n (scaling limit of the random walk).

Nowhere differentiability: W_t is everywhere continuous, but nowhere differentiable (almost surely). Variation: total variation is infinite, quadratic variation [W]_t = t (finite!). [W,W]_t = t — “key fact” underpinning Ito’s lemma.

Formulas

Ito's lemma is the stochastic analogue of the chain rule for differentiation. Due to the nonzero quadratic variation of Brownian motion, there appears an additional "correction" second-order term.

The integral ∫₀^T f_t dW_t for adapted processes fₜ. It cannot be defined pointwise (W_t is nowhere differentiable). It is defined as the L²-limit of step processes.

Properties: Martingale: E[∫₀^T f dW] = 0. Ito isometry: E[(∫₀^T f dW)²] = E[∫₀^T f² dt].

Let dX = b dt + σ dW, f ∈ C²(ℝ). Then: df(X_t) = f'(X_t) dX_t + (1/2)f''(X_t) d[X,X]_t = [f'(X_t)b + (1/2)f''(X_t)σ²] dt + f'(X_t)σ dW_t.

Definitions

IV

— the value of $\sigma$ at which the BS formula matches the market price. IV $\neq$ realized volatility. Volatility smile: IV depends on strike $K$ — violation of BS assumptions. In reality: IV is higher for OTM options (tail risk). Volatility surfa...

Formulas

Mathematical financial theory uses stochastic calculus for derivative pricing, risk management, and optimal investing. Girsanov's theorem and the Black-Scholes formula are central results.

The problem: In the real world, a stock grows at a rate μ > r (risk-free rate). For pricing, a "risk-neutral" measure Q is needed.

Girsanov's theorem: When switching measure from P to Q via $dQ/dP = e^{-\theta W_T - \theta^2 T/2}$: $\tilde{W}_t = W_t + \theta t$ is a Q-Brownian motion. For GBM with μ: $dS = \mu S\,dt + \sigma S\,dW \rightarrow dS = r S\,dt + \sigma S\,d\tilde{W}$ (at $\theta=(\mu - r)/\sigma$).

Pricing: $V_0(F) = e^{-rT} \mathbb{E}^Q[F_T]$ — price of a derivative with payoff $F_T$. This follows from the absence of arbitrage: the discounted price is a Q-martingale.

Definitions

$I(\theta) = -E[\frac{\partial^2}{\partial \theta^2} \log f(X; \theta)]$

— "expected information" in one observation. The greater the curvature of the log-likelihood, the more information. Information inequality: $\operatorname{Var}(\hat{\theta}) \geq 1/(nI(\theta))$—lower limit of the variance. For normal $N(\mu, \sigma...

Formulas

Asymptotic statistics studies the behavior of estimators as n → ∞. Consistency, asymptotic normality, and the delta method are the main tools for analyzing estimator properties.

Weak consistency: $\hat{\theta}_n \to_P \theta_0$ as $n\to\infty$. A sufficient condition: $\operatorname{Bias}(\hat{\theta}_n)\to 0$ and $\operatorname{Var}[\hat{\theta}_n] \to 0$. Mean square consistency (MSE→0) implies weak consistency.

Strong consistency: $\hat{\theta}_n \to_{a.s.} \theta_0$. The MLE is strongly consistent under regular conditions—this follows from the strong law of large numbers applied to the log-likelihood.

Invariance of MLE: If $\hat{\theta}$ is the MLE for $\theta$, then $g(\hat{\theta})$ is the MLE for $g(\theta)$ (for any function $g$). This follows as a consequence of the definition via likelihood maximization.

Formulas

Cramer-Rao bound establishes the lower limit for the variance of unbiased estimators. An asymptotically efficient estimator attains this bound as n → ∞.

Definition: I(θ) = E[(∂ log f(X;θ)/∂θ)²] = -E[∂² log f(X;θ)/∂θ²]. Fisher information measures the "steepness" of the likelihood function—the higher it is, the more precisely θ can be estimated.

Additivity: For n i.i.d. observations: Iₙ(θ) = n·I(θ). Matrix form: I(θ)ᵢⱼ = -E[∂² log f/∂θᵢ∂θⱼ]—the Fisher information matrix.

Examples: N(μ,σ²) with σ² known: I(μ)=1/σ². Poisson(λ): I(λ)=1/λ. Bernoulli(p): I(p)=1/(p(1-p)). Exp(λ): I(λ)=1/λ².

Definitions

Breakdown point ($\epsilon^*$)

— maximum proportion of spoiled data for which the estimator remains bounded. $\bar{X}$: $\epsilon^* = 1/n \approx 0$ (a single outlier can cause it to diverge). Median: $\epsilon^* = 0.5$ (theoretically the best). Trimmed mean ($\alpha$-trimmed): $...

Formulas

Classical estimators (mean, MLE) are sensitive to outliers. Robust statistics develops estimators resilient to data contamination while maintaining high efficiency in the normal case.

Breakdown Point (BP): The minimal proportion of contaminated data at which an estimator breaks down. $\bar{X}$: BP = $1/n \to 0$. Median: BP = $0.5$. $S^2$: BP = $1/n$. MAD $= \text{median}|X_i - \text{median}|$: BP = $0.5$.

Influence Function (Hampel, 1974): $IF(x; T, F) = \lim_{\varepsilon \to 0} \frac{T((1-\varepsilon)F + \varepsilon \delta_x) - T(F)}{\varepsilon}$. For $\bar{X}$: $IF(x) = x - \mu$ — unbounded. For the median: $IF(x) = \frac{\text{sign}(x-\mu)}{2f(\mu)}$ — bounded. Gross-error sensitivity: $\gamma...

Definition: M-estimator $\hat{\theta} = \arg\min_\theta \sum_i \rho(X_i - \theta)$, or solution to $\sum_i \psi(X_i - \theta) = 0$, $\psi = \rho'$. Mean: $\rho(r) = r^2/2$. Median: $\rho(r) = |r|$. MLE: $\rho(r) = -\log f(r)$.