Generating Functions and Characteristic Functions

Generating functions are a powerful tool for working with distributions of the sum of independent random variables. The characteristic function always exists and uniquely determines the distribution.

Moment Generating Function (MGF)

Definition: $M_X(t) = E[e^{tX}] = \sum E[X^n] t^n / n!$ (Taylor series). If it exists in a neighborhood of $t=0$: $E[X^n] = M_X^{(n)}(0)$. The MGF uniquely determines the distribution.

For the sum of independents: $M_{X+Y}(t) = M_X(t)\cdot M_Y(t)$. This turns convolutions into multiplications.

Examples: Poisson$(\lambda)$: $M(t) = \exp(\lambda(e^t-1))$. $N(\mu,\sigma^2)$: $M(t) = \exp(\mu t + \sigma^2 t^2 / 2)$. Sum of normal distributions: $M_{X+Y} = \exp((\mu_1+\mu_2)t + (\sigma_1^2+\sigma_2^2)t^2/2) \to N(\mu_1+\mu_2, \sigma_1^2+\sigma_2^2)$. ✓

Probability Generating Function (PGF)

For discrete $X \geq 0$: $G_X(z) = E[z^X] = \sum_k P(X=k)z^k$. $P(X=k) = G^{(k)}(0)/k!$. $E[X] = G'(1)$, $Var[X] = G''(1) + G'(1) - (G'(1))^2$.

Sum of a random number: $N \sim$ PMF, $X_1,X_2,\ldots$ i.i.d. $S_N = X_1+\ldots+X_N$. $G_{S_N}(z) = G_N(G_X(z))$. Powerful formula—"the total probability through PGF".

Characteristic Function

Definition: $\varphi_X(t) = E[e^{itX}] = \int e^{itx}f(x)dx$ (Fourier transform of the density). Always exists, $|\varphi(t)| \leq 1$.

Uniqueness: $\varphi_X \equiv \varphi_Y \Leftrightarrow$ $X$ and $Y$ have the same distribution. Inversion theorem: $f(x) = (1/2\pi) \int e^{-itx}\varphi(t)dt$ (inverse Fourier).

Continuity theorem (Lévy): $X_n \to X$ in distribution $\Leftrightarrow \varphi_{X_n}(t) \to \varphi_X(t)$ pointwise.

Exercise: (a) MGF for $U[0,1]$: $M(t) = (e^t - 1)/t$. Compute $E[X]$ and $E[X^2]$ via derivatives. (b) Sum of $n$ i.i.d. Poisson$(\lambda)$: show via MGF that the sum $\sim$ Poisson$(n\lambda)$. (c) What is the characteristic function of the Cauchy distribution? Why does Cauchy lack an MGF?

MGF and Computing Distribution Moments

MGF is a systematic tool for calculating moments of all orders. For normal $N(\mu,\sigma^2)$: $M(t) = \exp(\mu t + \sigma^2 t^2/2)$. Then $M'(0) = \mu = E[X]$, $M''(0) = \sigma^2 + \mu^2 = E[X^2]$, whence $Var[X] = \sigma^2$. For gamma $\Gamma(\alpha,\beta)$: $M(t) = (\beta/(\beta-t))^\alpha$ for $t < \beta$. $E[X] = \alpha/\beta$, $E[X^2] = \alpha(\alpha+1)/\beta^2$.

Cumulants: The logarithm of the MGF $K(t) = \ln M(t)$ is the generating function of cumulants. $K^{(n)}(0) = \kappa_n$—the $n$-th cumulant. First cumulants: $\kappa_1 = E[X]$ (mean), $\kappa_2 = Var[X]$ (variance), $\kappa_3 = \mu_3$ (third central moment), $\kappa_4 = \mu_4 - 3\sigma^4$ (excess kurtosis). Key property: cumulants of the sum of independent RVs add up: $\kappa_n(X+Y) = \kappa_n(X) + \kappa_n(Y)$.

Heavy Tails and Absence of MGF

For distributions with heavy tails, MGF does not exist for any $t > 0$. Cauchy distribution: $f(x) = 1/(\pi(1+x^2))$—symmetric, but $E[|X|] = \infty$. MGF does not exist. Characteristic function: $\varphi(t) = e^{-|t|}$. The sum of $n$ i.i.d. Cauchy has the same distribution with scale $n$—"LLN does not work"!

Pareto distribution: $P(X>x) = (x_0/x)^\alpha$ for $x \geq x_0$. For $\alpha \leq 1$: $E[X] = \infty$. For $\alpha \leq 2$: $Var[X] = \infty$. Models wealth ($\alpha \approx 1.5$ for Forbes 400), city size (Zipf's law), volume of internet traffic. MGF does not exist. For heavy tails, Laplace transform or survival function is used.

Generating Functions and Recurring Event Chains

Probability generating functions are especially powerful in the analysis of branching processes. Galton–Watson process: Each individual produces $X$ offspring (i.i.d.). $Z_n$—population of the $n$-th generation. $G_{Z_n}(z) = G_{n-fold\ composition}(z)$. Extinction probability $q = P(Z_n \to 0)$—the smallest non-negative root $q = G(q)$. For $E[X] \leq 1$: $q = 1$ (extinction inevitable). For $E[X] > 1$: $q < 1$. This is a mathematical model for the spread of genes, epidemics, and nuclear chain reactions.

Characteristic Function: Applications and Theorems

The characteristic function $\varphi_X(t) = E[e^{itX}]$ exists for any distribution. For normal $N(\mu,\sigma^2)$: $\varphi(t) = \exp(i\mu t - \sigma^2 t^2 / 2)$. For Poisson Poisson$(\lambda)$: $\varphi(t) = \exp(\lambda(e^{it}-1))$. For Cauchy Cauchy$(0,1)$: $\varphi(t) = e^{-|t|}$.

Lévy continuity theorem: $X_n \to X$ in distribution $\Leftrightarrow \varphi_{X_n}(t) \to \varphi_X(t)$ for each $t$. This allows proving the CLT and other limit theorems via characteristic functions. Characteristic function method: "fix $t$, compute $\varphi_X(t)$ for the sum, take the limit, recognize the distribution".

Proof of CLT via characteristic functions: Let $X_i$ i.i.d. with $E[X]=0$, $Var[X]=\sigma^2$. Then $\varphi_{X_i}(t) = 1 - \sigma^2 t^2 / 2 + o(t^2)$. For standardized sum $S_n = (X_1+\ldots+X_n)/(\sigma\sqrt{n})$: $\varphi_{S_n}(t) = (1 - t^2/2n + o(t^2/n))^n \to e^{-t^2/2} = \varphi_{N(0,1)}(t)$. QED.

MGF for Risk Assessment: Value-at-Risk and CVaR

In risk management, Value-at-Risk (VaR) at level $\alpha$: quantile $Q(\alpha)$ of losses. Conditional Value-at-Risk (CVaR, or Expected Shortfall): $E[X|X \geq VaR(\alpha)]$—expected losses above the threshold. CVaR is linked to the distribution tail and is calculated via Laplace transform/MGF for subexponential distributions. In insurance: pure net premium $= E[X]$, standard deviation loading $= \lambda \cdot \sigma(X)$. Both are computed via MGF.

Numerical Example: Moments through MGF

Problem: $X \sim Exp(\lambda=3)$. Find $E[X]$ and $Var[X]$ via the moment generating function.

Step 1: MGF for the exponential distribution: $M(t) = \lambda/(\lambda-t) = 3/(3-t)$, defined at $t < 3$.

Step 2: $E[X] = M'(0)$. Differentiate: $M'(t) = 3 \cdot 1/(3-t)^2 \rightarrow M'(0) = 3/9 = 1/3 \approx 0.333$.

Step 3: $E[X^2] = M''(0)$. Differentiate once more: $M''(t) = 6/(3-t)^3 \rightarrow M''(0) = 6/27 = 2/9 \approx 0.222$.

Step 4: $Var[X] = E[X^2] - (E[X])^2 = 2/9 - 1/9 = 1/9 \approx 0.111$, $\sigma = 1/3$. Matches the formula: $Exp(\lambda)$ has $E[X]=1/\lambda=1/3$ and $Var[X]=1/\lambda^2=1/9$. ✓ $VaR_{0.95} = -\ln(0.05)/3 \approx 1.0$—tail quantile, estimated via MGF integral.