Probability Theory Basics for Engineers

Probability theory is the mathematical framework for reasoning about uncertainty. Every prediction, every decision under risk, every machine learning model rests on probability. Yet most engineers and scientists who use probability have only a hazy intuition for what it means — they remember formulas without understanding the structure.

This article covers what a probability is, the formal structure (probability spaces), random variables, the most important distributions, conditional probability and Bayes' theorem, expectation and variance, the central limit theorem, and applications in engineering and machine learning.

What probability is

Probability is a measure of the likelihood of an event. We assign a number between 0 and 1 to each event:

• 0 means the event is impossible.
• 1 means the event is certain.
• 0.5 means equally likely as not.

This is the intuition. The formal definition is more careful.

A probability space is a triple $(\Omega ,\mathcal{F},P)$ where:

• $\Omega$ is the sample space — the set of all possible outcomes.
• $\mathcal{F}$ is a collection of events — subsets of $\Omega$ that we can assign probabilities to.
• $P$ is the probability measure — a function from $\mathcal{F}$ to $[0,1]$ satisfying:
  • $P(\Omega )=1$.
  • $P(A)\ge 0$ for every event $A$.
  • For disjoint events $A_1,A_2,\dots$: $P(A_1\cup A_2\cup \cdots )=P(A_1)+P(A_2)+\cdots$.

This is the Kolmogorov axiomatization (1933), which gave probability theory its current rigorous foundation.

In practice, you rarely think about probability spaces explicitly. You work with random variables, distributions, and probabilities of events.

Discrete vs continuous probability

Discrete probability handles situations with countable outcomes. Coin flips, dice rolls, customer counts, page views.

Continuous probability handles situations with uncountably many outcomes. Measurements, times, weights, lengths.

The distinction matters because discrete distributions assign positive probability to individual outcomes (e.g., $P(\text{coin = heads})=0.5$), while continuous distributions assign probability zero to any individual outcome. For continuous variables, you talk about probabilities of ranges: $P(X<5)$ or $P(2\le X\le 5)$.

Continuous probability is described by a probability density function (PDF). The PDF $f(x)$ has the property that

$P(a\le X\le b)={\int }_a^{b}f(x)dx$

The PDF is not itself a probability — it is a density. The integral of the PDF over an interval gives the probability of falling in that interval.

Random variables

A random variable is a function from the sample space to the real numbers. In English: a quantity whose value depends on a random outcome.

Examples:

• $X$ = number of heads in 10 coin flips (discrete, takes values 0 to 10).
• $Y$ = height of a randomly selected adult (continuous, takes values in roughly 50–250 cm).
• $Z$ = time until a server fails (continuous, takes positive real values).

Random variables are usually denoted by capital letters ($X$, $Y$, $Z$). Specific realizations (observed values) are denoted by lowercase ($x$, $y$, $z$).

Common distributions

A small number of distributions appear repeatedly in applications. Memorize their forms and properties.

Discrete distributions:

Bernoulli$(p)$ — single trial with two outcomes. $P(X=1)=p$, $P(X=0)=1-p$. Example: single coin flip.

Binomial$(n,p)$ — number of successes in $n$ independent Bernoulli$(p)$ trials. $P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$. Example: number of heads in 10 coin flips.

Poisson$(\lambda )$ — number of events in a fixed interval, when events occur independently at rate $\lambda$. $P(X=k)=\frac{{\lambda }^k{e}^{-\lambda }}{k!}$. Example: number of customers arriving in an hour.

Geometric$(p)$ — number of trials until the first success. $P(X=k)=(1-p{)}^{k-1}p$. Example: number of coin flips until the first heads.

Continuous distributions:

Uniform$(a,b)$ — equally likely to be anywhere in $[a,b]$. PDF: $f(x)=\frac{1}{b-a}$ for $x\in [a,b]$, zero elsewhere.

Normal (Gaussian) $\mathcal{N}(\mu ,{\sigma }^2)$ — the bell curve. PDF: $f(x)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-(x-\mu {)}^2/(2{\sigma }^2)}$. Mean $\mu$, variance ${\sigma }^2$. The single most important distribution in probability and statistics.

Exponential$(\lambda )$ — waiting time between events of a Poisson process. PDF: $f(x)=\lambda e^{-\lambda x}$ for $x\ge 0$. Memoryless property.

Beta$(\alpha ,\beta )$ — distribution on $[0,1]$, useful for modeling probabilities. PDF: $f(x)=\frac{x^{\alpha -1}(1-x{)}^{\beta -1}}{B(\alpha ,\beta )}$.

Each distribution has parameters that determine its shape. Each has a mean, variance, and other moments computable from the parameters.

Conditional probability and Bayes' theorem

The conditional probability of $A$ given $B$ is

$P(A|B)=\frac{P(A\cap B)}{P(B)}$

assuming $P(B)>0$. In words: the probability of $A$, given that we know $B$ has occurred.

Bayes' theorem rewrites this:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

Why this matters: it lets you reverse conditional probabilities. If you know $P(B|A)$ and you want $P(A|B)$, Bayes' theorem gets you there.

Example 1 — the classic medical test: A disease affects 1 in 1000 people. A test for the disease is 99% accurate (both sensitivity and specificity). If a person tests positive, what is the probability they have the disease?

Intuition says: about 99%, since the test is 99% accurate. Bayes' theorem says otherwise.

Let $D$ = "has disease," $T$ = "tests positive."

$P(D)=0.001$. $P(T|D)=0.99$ (true positive rate). $P(T|\neg D)=0.01$ (false positive rate).

By the law of total probability: $P(T)=P(T|D)P(D)+P(T|\neg D)P(\neg D)=0.99\cdot 0.001+0.01\cdot 0.999=0.00099+0.00999=0.01098$.

Bayes: $P(D|T)=\frac{P(T|D)P(D)}{P(T)}=\frac{0.99\cdot 0.001}{0.01098}\approx 0.090$.

So if you test positive, the probability you actually have the disease is about 9%, not 99%. The 99% accuracy of the test is overwhelmed by the very low prior probability of the disease.

This is the base rate fallacy, and it is one of the most consequential cognitive errors in medical, judicial, and security contexts.

Expectation and variance

The expected value (or mean) of a random variable is a weighted average:

Discrete: $E[X]={\sum }_{x}x\cdot P(X=x)$. Continuous: $E[X]={\int }_{-\infty }^{\infty }x\cdot f(x)dx$.

Example 2: Expected value of a die roll.

$E[X]=1\cdot \frac{1}{6}+2\cdot \frac{1}{6}+\cdots +6\cdot \frac{1}{6}=\frac{1+2+3+4+5+6}{6}=3.5$.

(Note: 3.5 is not a possible outcome. Expected value is an average, not a typical value.)

Variance measures the spread of a distribution:

$\text{Var}(X)=E[(X-E[X]{)}^2]=E[X^2]-E[X{]}^2$

The standard deviation is the square root of variance: ${\sigma }_X=\sqrt{\text{Var}(X)}$.

Linearity of expectation: $E[aX+bY]=aE[X]+bE[Y]$ for any random variables $X$, $Y$ and constants $a$, $b$. This holds whether or not $X$ and $Y$ are independent.

Variance is not linear: $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)+2\text{Cov}(X,Y)$. For independent variables, the covariance is zero and variances add.

The central limit theorem

The single most important theorem in probability. Plain version:

The sum (or average) of a large number of independent random variables, regardless of their individual distributions, is approximately normally distributed.

Formally: if $X_1,X_2,\dots ,X_n$ are independent and identically distributed (i.i.d.) random variables with mean $\mu$ and variance ${\sigma }^2$, then for large $n$:

$\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}\to \mathcal{N}(0,1)$

in distribution, where ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ is the sample mean.

Why this matters. Most real-world quantities are sums or averages of many smaller random influences — measurement errors, financial returns, biological traits, manufacturing variations. The central limit theorem explains why the normal distribution appears everywhere: not because the universe loves it, but because sums of independent random things approach it.

The central limit theorem is also the foundation of statistical inference. Confidence intervals, hypothesis tests, and most of practical statistics rely on the asymptotic normality it guarantees.

Applications

Engineering reliability. The exponential distribution models the time-to-failure of components with constant failure rate. Compound systems (multiple components in series or parallel) have failure-time distributions computable from the components' distributions.

Communication theory. Shannon's information theory uses probability to quantify information. Channel capacity, error rates, and compression are all probability problems.

Machine learning. Classification outputs are probability distributions. Bayesian methods model uncertainty in parameters and predictions. Generative models learn the probability distributions of data.

Finance. Asset returns are modeled as random variables. Portfolio variance is computed from the covariance matrix of returns. The Black-Scholes option pricing formula assumes returns are normally distributed (lognormal prices).

Quality control. Statistical process control uses probability distributions of manufacturing measurements to detect process drift before quality problems become severe.