Cheatsheet

Optimal Control

All topics on one page

5modules

15articles

1definitions

18formulas

Calculus of Variations

The Lagrange problem, the Euler–Lagrange equation, and classical problems

Functionals and the Euler–Lagrange Equation

Statement of a Variational Calculus Problem → Euler–Lagrange Equation → Classic Examples → Numerical Example: Minimizing $\int_0^1 (y^2 + y'^2)\,dx$, $y(0) = 1$, $y(1) = 0$ → Extensions → Real Applications

Formulas

$\frac{\partial F}{\partial y} - \frac{d}{dx}\left(\frac{\partial F}{\partial y'}\right) = 0$.

·Architecture and Engineering. The shape of arches and suspension bridges arises from minimizing potential energy—the catenary $y = a\cdot\cosh(x/a)$. The Golden Gate suspension bridge, cathedral ar...
·Optics. Fermat’s principle gives the laws of reflection and refraction. All geometric optics is a consequence of the variational principle.
·Machine Learning. Model training reduces to minimization of an error functional—this is a “discrete” counterpart of the variational problem. Regularization (e.g., Tikhonov) adds a penalty term $\in...
·Finance. Optimal trajectories of consumption and investment (the Merton model) are found as extrema of the respective expected utility functional.

Imagine stretching a rope between two nails: what shape will it take? Or: along which curve will a ball slide down in minimal time? In standard differential calculus, we seek a number $x$ that minimizes a function $f(x)$. Here, the unknown is an entire function $y(x)$, and we seek to minimize the...

Problem: Find a function $y(x)$ on the interval $[a, b]$ which minimizes the functional $J[y] = \int_a^b F(x, y, y')\,dx$ under boundary conditions $y(a) = y_a$, $y(b) = y_b$.

Let us clarify the notation. $F(x, y, y')$ is the Lagrangian, the “density” of the quantity of interest: for example, arc length, travel time, or action. $x$ is the independent variable (often position or time), $y$ is the sought function, $y' = dy/dx$ is its derivative. The integral $J[y]$ sums ...

In physics, typically $F = T − V$ (kinetic energy minus potential energy), and $J$ is called the action. In optics, $F = n(x, y)\sqrt{1 + y'^2}$, where $n$ is the refractive index, and $J$ is the optical path.

Second Order Conditions and Sufficient Conditions for Extremum

Second Variation → Field of Extremals and the Weierstrass Theorem → Connection with Hamiltonian Mechanics → Numerical Example: $\min \int_0^1 (y^2 + {y'}^2)\,dx,\ y(0) = 1,\ y(1) = 2$ → Real Applications

Formulas

Weierstrass E-function: $E(x, y, p, y') = F(x, y, y') - F(x, y, p) - (y' - p)\cdot \frac{\partial F}{\partial y'}(x, y, p)$.

·Mechanical Engineering. The shape of gear teeth is chosen to ensure smooth transfer of forces—this is a variational problem with second order conditions for smoothness of the envelope.
·Financial Mathematics. The optimal Merton strategy (consumption + investment) is verified for sufficiency via analysis of HJB and second order conditions for the value function.
·Production Management. Holt-Winters models for optimal production scheduling use quadratic functionals—the Legendre condition ensures convexity in control, and absence of conjugate points guarantee...
·Numerical Optimization. Newton-type algorithms for functionals use the Hessian $\delta^2 J$—if it is positive definite on admissible directions, the Newton step converges correctly to a minimum.

The Euler-Lagrange equation is only a necessary condition, analogous to the equality $f'(x) = 0$ in regular analysis. But $f'(x) = 0$ vanishes the same way at a minimum, at a maximum, and at a saddle point. To distinguish a true minimum, second order conditions are needed. In the calculus of vari...

Analogous to the Taylor expansion $f(x + \varepsilon) \approx f(x) + \varepsilon f'(x) + \frac{\varepsilon^2}{2} f''(x)$, for a functional $J[y + \varepsilon\eta]$ one has the expansion:

$ J[y + \varepsilon\eta] = J[y] + \varepsilon \cdot \delta J[y, \eta] + \frac{\varepsilon^2}{2} \cdot \delta^2 J[y, \eta] + O(\varepsilon^3). $

When $\delta J = 0$ (EL is satisfied), the sign of $\delta^2 J$ determines whether it is a minimum.

Variational Principles in Mechanics and Physics

Principle of Least Action → Examples of Physical Systems → Hamiltonian Mechanics → Noether’s Theorem → Real Applications

·Invariance in time $\rightarrow$ law of conservation of energy.
·Invariance under spatial translations $\rightarrow$ law of conservation of momentum.
·Invariance under rotation $\rightarrow$ law of conservation of angular momentum.
·Robotics. The equations of motion for a multi-link manipulator (5–7 degrees of freedom) are derived via the Lagrangian—this is much simpler than spelling out the individual interaction forces. Inve...
·Molecular Dynamics. Protein simulations (10⁵–10⁶ atoms) integrate the Hamiltonian equations with a step of ~1 fs. Symplectic integrators (Verlet) preserve energy over long times—critical for biophy...
·Financial Physics (econophysics). Models of market dynamics use the Lagrangian formalism to describe the “motion” of prices under the influence of the “potentials” of supply and demand.

"Nature prefers simplicity"—this aphorism is embodied in the principle of least action, one of the deepest principles of physics. Instead of "solving" the equations of motion step by step for each point of a trajectory, nature "chooses" the entire trajectory at once—the one along which the action...

Action: $S[q] = \int_{t_1}^{t_2} L(q, \dot{q}, t) \, dt$, where $L = T - V$ is the Lagrangian of the system (kinetic energy minus potential energy). $q = (q_1, ..., q_n)$ are the generalized coordinates (e.g., angles, lengths), $\dot{q}$ are the generalized velocities.

Hamilton's Principle: The actual trajectory of the system between fixed states $q(t_1)$ and $q(t_2)$ is a stationary point of the action: $\delta S = 0$.

$ \frac{d}{dt} \left( \frac{\partial L}{\partial \dot{q}_i} \right) - \frac{\partial L}{\partial q_i} = 0, \quad i = 1, ..., n. $

Pontryagin’s Maximum Principle

Optimal control in continuous time, the Hamiltonian, and adjoint variables

Formulation of the Optimal Control Problem

Standard Optimal Control Problem → Pontryagin Maximum Principle → The Difference from Calculus of Variations → Numerical Example: Minimum-Time Problem → Real-World Applications

Formulas

Bolza Problem: $\max J = \varphi(x(T), T) + \int_0^T L(x(t), u(t), t)\,dt$ subject to:

·State equation: $\dot{x} = f(x(t), u(t), t)$, $x(0) = x_0$.
·Control: $u(t) \in U \subset \mathbb{R}^m$ is the admissible set (often compact).
·Boundary conditions: $\psi(x(T), T) = 0$ (or $x(T)$ is free).
·Lagrange problem: $\varphi \equiv 0$ (only an integral criterion).
·Mayer problem: $L \equiv 0$ (terminal only).
·Minimum-time problem: $J = T$ (minimize time).
·Aerospace industry. The Apollo Guidance Computer (1969) used a simplified form of the PMP to optimize the lunar lander trajectory. Modern SpaceX Falcon 9 rockets solve the soft-landing problem in r...
·Robotics. Minimum-time motion of a manipulator between two positions is an optimal control problem with constraints on torque. Bang-bang control is often optimal but requires smoothing for mechanic...
·Autonomous transportation. Tesla Autopilot, Waymo solve trajectory planning tasks taking into account limits on acceleration and turning angle. Modifications of PMP and MPC (Model Predictive Contro...
·Epidemiology. The optimal vaccination/lockdown strategy during COVID was modeled as an optimal control problem for the SEIR model with control being the “stringency” of restrictions.

In the calculus of variations, the sought-for function $y(x)$ is smooth and unconstrained. But imagine a spacecraft with engine thrust $u(t)$: its magnitude cannot exceed the maximum thrust, it can be turned on and off abruptly. Classical calculus is powerless here—new machinery is needed. Optima...

Bolza Problem: $\max J = \varphi(x(T), T) + \int_0^T L(x(t), u(t), t)\,dt$ subject to:

Explanation. $x(t) \in \mathbb{R}^n$ is the state of the system (position, velocity, fuel remaining). $u(t)$ is the control (thrust, rudder angle, tax rate). $L$ is instantaneous utility/cost, $\varphi$ is terminal reward/penalty, $f$ is the law of system evolution.

Unlike the calculus of variations, $u(t)$ may be discontinuous and may take values only from $U$. For $U = [u_{\min}, u_{\max}]$ the optimum often lies on the boundary—this gives the characteristic "bang-bang" control.

Maximum Principle: Proof and Applications

Idea of the Proof → Euler Condition as a Special Case → Economic Applications → Engineering Applications

·Optimal Landing. Falcon 9: the problem is to minimize fuel expenditure subject to constraints on angle of attack, thrust, final velocity. Solved via convex relaxation of PMP in real time.
·HVAC Systems. Optimum building air conditioning schedule (minimization of energy use while maintaining comfort temperature)—a classic optimal control problem.

The PMP (Pontryagin Maximum Principle) is a nontrivial result: its proof does not reduce to simple integration by parts as with the Euler–Lagrange (EL) equation. Pontryagin and his students (Boltyansky, Gamkrelidze, Mishchenko) constructed the proof via “needle variations” and the theorem of sepa...

Needle variations. Instead of smooth perturbations (as in EL), we consider a "needle": on a short interval $[t_0, t_0 + \varepsilon]$, we replace $u^*(t)$ with any admissible $v \in U$, leaving $u^*$ for the remainder of the time. This causes a jump in the state $x(t)$, and, accordingly, in the o...

Cone of variations. The collection of all such perturbations in the limit as $\varepsilon \to 0$ yields the attainable cone in the state space. If $u^*$ is optimal, the attainable cone and the direction of improvement of $J$ must be on “opposite sides” of some hyperplane—otherwise, there would ex...

Separation of cones. The normal to the separating hyperplane is the vector $\psi(t_0)$. The condition that a needle variation does not improve $J$ is equivalent to $H(x^*(t_0), v, \psi(t_0), t_0) \leq H(x^*(t_0), u^*(t_0), \psi(t_0), t_0)$ for all $v \in U$. This is precisely the maximum condition.

Problems with Finite and Infinite Horizon

Finite Horizon Problem → Infinite Horizon Problem → Linear-Quadratic Problems (LQR) → Numerical Example: LQR for a Scalar System → LQR for the Inverted Pendulum → Real Applications

Formulas

Standard form: max ∫₀^∞ e^{−ρt}·F(x, u) dt subject to ẋ = f(x, u), x(0) = x₀.

·Aircraft autopilot. Each of the loops (pitch, roll, yaw) is designed as LQR around the flight regime. For different speeds/altitudes a set of K is computed, between which “gain scheduling” occurs.
·Segway/two-wheeled robot stabilization. Standard LQR for the model of an inverted pendulum on wheels.
·Power plant control. Maintaining stable frequency (50 Hz) under load fluctuations — LQR control of turbines.
·Finance. The optimal trading strategy with quadratic market impact costs (Almgren–Chriss) reduces to LQR — control = trading speed.

Engineering problems (rocket landing, satellite maneuvering) usually have a finite horizon — the criterion is tied to the end moment. Economic problems (consumption, growth) are often infinite — we want to optimize welfare “forever.” These two classes require different transversality conditions a...

The transversality condition depends on how the boundary conditions are set at moment T.

Case 1: x(0) and x(T) are fixed. The conjugate variables ψ(0), ψ(T) are determined from the solution of the boundary value problem (x, ψ).

Case 2: x(T) is free, T is fixed. Additional condition: ψ(T) = ∂φ/∂x(x(T), T). If there is no terminal penalty (φ = 0), then ψ(T) = 0.

Bellman’s Dynamic Programming

The principle of optimality, the Bellman equation, and the value function

Principle of Optimality and the Bellman Equation

Bellman’s Principle of Optimality → Hamilton-Jacobi-Bellman Equation (HJB) → Verification Theorem → Numerical Example: LQR with Finite Horizon → Connection with the Maximum Principle → Real Applications

Formulas

$-\frac{\partial V^*}{\partial t} = \max_{u \in U} \left[ L(x, u, t) + \left(\frac{\partial V^*}{\partial x}\right)^\top f(x, u, t) \right]$.

·Maximum Principle: a boundary value problem (forward for $x$, backward for $\psi$) — efficient for a small number of states.
·HJB: PDE for $V^*(x, t)$ — efficient with complex constraints, but suffers from the curse of dimensionality for large $\dim(x)$.
·Finance. Merton’s problem (optimal consumption and investment) was solved explicitly via HJB in 1969: the policy “invest a fixed share $\pi^* = \frac{\mu - r}{\sigma^2 \gamma}$ in risky assets” was...
·Inventory management. The Scarf and Arrow-Harris models use HJB to find the optimal $(s, S)$ policy: “if inventory
lt;$ $s$, order up to $S$.”
·Advertising and marketing. Dynamic allocation of budget between advertising channels — HJB problem with empirically calibrated response models.
·Reinforcement Learning. The Bellman equation is the basis of Q-learning, DQN, AlphaGo. In each episode, the agent updates $V$ (or $Q$) according to the discrete-time version of the HJB.

Richard Bellman in the 1950s proposed a radically different perspective on optimization: not “find the entire trajectory at once” (as in the Maximum Principle), but “solve the problem recursively — from right to left.” At its core lies a deep principle: the optimal plan “remembers” only the curre...

Formulation: “An optimal policy has the property that, whatever the initial state and initial decision are, the remaining decisions constitute an optimal policy with regard to the state resulting from the first decision.”

More simply: any “tail part” of an optimal trajectory is itself optimal for the corresponding subproblem.

Formally. Define the optimal value function: $ V^*(x, t) = \max_{u(\cdot)\ \text{on}\ [t,T]} \left[ \int_t^T L(x, u, s)\ ds + \varphi(x(T)) \right], \quad \text{where}\ x(t) = x. $

Discrete Dynamic Programming and Numerical Methods

Discrete Dynamic Programming → Numerical Example: The Knapsack-in-Time Problem → Curse of Dimensionality → Approximate DP → Direct Trajectory Optimization Methods → Comparison of Approaches → Real-World Applications

Definitions

This is the “curse of dimensionality”: — a fundamental barrier for tabular DP. Overcoming it is a key subject of modern methods (approximate DP, reinforcement learning).

Formulas

Complexity: $O(T \cdot |S| \cdot |U|)$. For $T = 100$, $|S| = 100$, $|U| = 10$: $10^5$ operations—instantaneous.

Approach	Advantages	Disadvantages
DP (VFI/PFI)	Global optimum, feedback $u^*(x, t)$	Curse of dimensionality
PMP + boundary value problem	Efficient for small dim	Local optimum, no feedback
Direct methods (NLP)	Large problems, constraints	Local optimum, open-loop control
Approximate DP / RL	Complex environments, neural nets	Requires delicate tuning, no guarantees

·Mars Curiosity & Perseverance. Rover path planning: discrete DP on altitude grid (DEM maps from orbit), cost—energy + risk of overturning.
·Amazon Inventory Management. Multi-level model (supplier → warehouse → regional center → customer)—gigantic DP with an approximate value function.
·Self-Driving Car. MPC (Model Predictive Control) with horizon 3–5 seconds: at each step, solve an NLP to select trajectory. Solution speed 10–50 Hz.
·Power Grids. Optimal control of EV charging in a smart grid—stochastic DP with hundreds of thousands of agents, solved via approximate methods.

Analytical solutions to optimal control problems are possible only for special structures (LQR, the Ramsey problem with Cobb-Douglas, the Merton problem). The majority of practical problems—nonlinear, high-dimensional, with discrete decisions—must be solved numerically. There exist two major fami...

Problem: $\max \sum_{t=0}^{T-1} r(x_t, u_t) + V_T(x_T)$ subject to $x_{t+1} = f(x_t, u_t)$, $x_t \in S$, $u_t \in U$.

Here $S$ is the set of states (for example, $|S| = 100$), $U$ is the set of actions, $r$ is the immediate reward.

Backward Induction Algorithm. 1. Initialization: $V_T(x)$ is given for all $x \in S$ (terminal value). 2. For $t = T-1, T-2, ..., 0$ and each $x \in S$: $ V_t(x) = \max_{u \in U} [r(x, u) + V_{t+1}(f(x, u))]. $ 3. Optimal policy: $u^*(t, x) = \arg\max_{u} [r(x, u) + V_{t+1}(f(x, u))]$.

DP in Economics: Accumulation, Resources, Growth

Basic Capital Accumulation Model (Stochastic Ramsey) → Numerical Example: Ramsey Model → Exhaustible Resource Problem (Hotelling Rule) → Neoclassical Growth Model: Quantitative Analysis (RBC) → Real Applications

Formulas

Solution through DP. V(s) = max_{q ≤ s} [u(q) + β·V(s − q)]. For u(c) = ln c, trial form: V(s) = A + B·ln(s).Calibration: α = 0.36, β = 0.99 (quarterly), δ = 0.025, σ = 1, shock z_{t+1} = ρ·z_t + ε_t, ρ = 0.95, σ_ε = 0.007.

·Log-GDP dispersion: model 1.7%, data 1.7%. ✓
·Investment dispersion / consumption dispersion: ~10, data ~6. Close.
·Output-labor correlation: 0.97, data 0.86. Overestimated.
·Central banks. The Fed, ECB, Bank of Russia use DSGE models (medium-scale—30-100 variables), solved via VFI or linearization, for forecasting and policy evaluation (Smets-Wouters, FRB/US, R-Quest).
·Insurance companies and pension funds. Long-term asset-liability management models—a stochastic DP with economic scenarios. Analogous to the Merton problem, over 20-50 years.
·Corporate finance. Capital investment decisions with irreversibility—the optimal stopping problem, a classical DP example (Dixit-Pindyck "Investment Under Uncertainty").
·Tax policy. Optimal taxation in Aiyagari models with heterogeneous agents—the gigantic DP with wealth distribution, solved by approximate methods (perturbation, EGM).

Dynamic programming is the standard language of modern macroeconomics and business strategy. Ramsey, Stokey-Lucas, RBC, Bewley, Aiyagari models—all are formulated through a recursive problem. Numerical methods (value function iteration, policy function iteration, Endogenous Grid Method) allow for...

Recursive problem: V(k) = max_{c ∈ [0, f(k)]} [u(c) + β·V(f(k) − c + k·(1 − δ))],

where k is capital, c is consumption, f(k) = k^α is production (Cobb-Douglas function), δ is the depreciation rate, β = 1/(1 + ρ) is the discount factor (ρ is the rate of time preference), u(c) = c^{1−σ}/(1 − σ) is the utility function with elasticity of substitution 1/σ.

Value Function Iteration method (VFI): 1. Discretize k onto a grid {k_1, ..., k_N}, for example N = 500 points. 2. Initialization: V_0(k_i) = 0 for all i. 3. In iteration n+1: V_{n+1}(k_i) = max_{c} [u(c) + β·V_n(k')], where k' = f(k_i) − c + k_i·(1 − δ). Interpolate V_n between grid points (line...

Linear Control and Stability

Linear systems, controllability, observability, and PID controllers

Linear Systems: Controllability and Observability

Linear Time-Invariant Systems → Controllability → Numerical Example: Double Integrator → Counterexample: Uncontrollable System → Observability → Canonical Forms → Luenberger Observer → Numerical Example: Observer for Double Integrator → Real-World Applications

Formulas

Matrix exponential: e^{At} = Σ_{k=0}^∞ (At)^k/k! — fundamental matrix. Computed via eigen-decomposition A = V·Λ·V⁻¹: e^{At} = V·diag(e^{λ_i·t})·V⁻¹.Example. Harmonic oscillator: A = [0, 1; −ω², 0]. Eigenvalues ±iω, e^{At} = [cos ωt, sin ωt/ω; −ω·sin ωt, cos ωt] — rotation in phase space.Controllability matrix: 𝓒 = [B | A·B | A²·B | ... | A^{n−1}·B] ∈ ℝ^{n × n·m}.Observability matrix: 𝒪 = [C; C·A; C·A²; ...; C·A^{n−1}] ∈ ℝ^{p·n × n}.

·GPS receivers. State x = (position, velocity, clock error), measurements are pseudoranges to satellites. The observer (extended Kalman filter) reconstructs position with 5–10 meter accuracy.
·Power systems. State estimation in SCADA: measurements of voltages and currents in network nodes → estimate the state of the whole network (thousands of variables) → dispatch control.
·Automotive electronics. Estimation of battery state of charge (SoC) in an electric car by current and voltage—Luenberger observer or Kalman filter.
·Biomedical devices. Continuous glucose monitors estimate “true” blood glucose concentration from readings of a subcutaneous sensor—an observability problem.

Before it is possible to “optimally” control a system, two fundamental questions must be answered: can the system, in principle, be brought to the desired state? And is it possible to determine the system's state from the available measurements? These issues—controllability and observability—are ...

Here x ∈ ℝⁿ is the state vector (position, velocity, temperature, currents), u ∈ ℝᵐ is the input (control), y ∈ ℝᵖ is the output (measurements). The matrices A (n×n), B (n×m), C (p×n), D (p×m) describe the system's physics. Often D = 0.

Matrix exponential: e^{At} = Σ_{k=0}^∞ (At)^k/k! — fundamental matrix. Computed via eigen-decomposition A = V·Λ·V⁻¹: e^{At} = V·diag(e^{λ_i·t})·V⁻¹.

Example. Harmonic oscillator: A = [0, 1; −ω², 0]. Eigenvalues ±iω, e^{At} = [cos ωt, sin ωt/ω; −ω·sin ωt, cos ωt] — rotation in phase space.

PID Control and Regulator Synthesis

Structure of the PID Regulator → PID Synthesis: Empirical Methods → Transfer Functions → Synthesis in Frequency Domain → Numerical Example: PID for G(s) = 1/(s·(s+1)·(s+5)) → Anti-windup and Practical Tricks → Real Applications

·Proportional (P). u_P = K_P·e. The greater the deviation, the stronger the response. Reduces the steady-state error, but does not eliminate it completely (there remains a “static error” at all time...
·Derivative (D). u_D = K_D·ė. Reacts to the rate of change of error → reduces overshoot, adds “damping”. Sensitive to measurement noise (the derivative of noise is huge). Often a filtered derivative...
·Phase Margin (PM) ≥ 45° — allowable phase variation before loss of stability.
·Gain Margin (GM) ≥ 6 dB (factor 2) — allowable gain variation.
·Bandwidth — frequency at which |G_closed| = −3 dB.
·Industrial Automation. Temperature controllers in furnaces (steel melting, ceramics firing), pressure (compressors, pumps), level (tanks, boilers) — almost always PID. Tuning strategy: “80% of task...
·Consumer Electronics. Thermostats for refrigerators, irons, multicookers — simplified PI. Modern boilers (Vaillant, Buderus) — PID with adaptive tuning.
·Aerospace. Internal loops of airplane autopilot (pitch, roll, yaw control) — cascade PID. External loops (trajectory, altitude) — more complex regulators on top of PID.
·Robotics. Control of each manipulator joint — PID (often with a feedforward term for gravity compensation). Level: 1 PID per joint, 5–7 joints per arm — total of 5–7 parallel PIDs.

PID regulator is the most widespread industrial controller: according to various estimates, more than 90% of control loops in industry use PID or its variations. It is simple, requires minimal knowledge of the object model, and when properly tuned provides acceptable quality for most tasks. Under...

where e(t) = r(t) − y(t) is the tracking error (the difference between the setpoint r and the actual output y), and K_P, K_I, K_D are the regulator coefficients.

Ziegler-Nichols Method (1942), variant 1 — step response. 1. Apply u(t) = 1 (step), record y(t). 2. From the plot estimate L (dead time — time until the reaction starts) and T (time constant — time to reach 63% of steady-state value). 3. Set: K_P = 1.2·T/L, T_I = 2·L (T_I = K_P/K_I), T_D = 0.5·L ...

Ziegler-Nichols Method, variant 2 — ultimate cycle. 1. Set K_I = K_D = 0, increase K_P until the system enters undamped oscillations. 2. Remember K_cr (critical gain) and T_cr (period of oscillations). 3. For PID: K_P = 0.6·K_cr, T_I = 0.5·T_cr, T_D = 0.125·T_cr.

Stability of Nonlinear Systems: The Lyapunov Method

Notions of Stability → The Direct Lyapunov Method → Searching for a Lyapunov Function → Examples → LaSalle's Invariance Principle (LaSalle's Theorem) → Control Lyapunov Functions (CLF) → Numerical Example: Stabilization of the Inverted Pendulum → Real Applications

Formulas

1. Linear system ẋ = A·x with A = [−1, 1; 0, −2]. Eigenvalues −1, −2 — stable.3. Nonlinear oscillator: ẍ + x + x³ = 0.Sontag's universal formula: u(x) = −[L_f V + √((L_f V)² + (L_g V)⁴)] / L_g V, if L_g V ≠ 0; otherwise u = 0. This formula guarantees V̇ < 0.

·Linear systems: V(x) = xᵀ·P·x with P > 0. V̇ = xᵀ·(AᵀP + PA)·x. For V̇ < 0, require Aᵀ·P + P·A = −Q < 0. This is the Lyapunov equation: for stable A and any Q > 0, there exists a unique solution P ...
·Mechanical systems: V = T + V_potential — the total energy, if it is positive definite.
·Systems with dissipation: often V = “energy”, V̇ ≤ 0.
·Power systems. Analysis of synchronous stability of generators after a short circuit: Lyapunov functions based on the kinetic + potential energy of generators. Used to determine the “critical clear...
·Robotics. Adaptive control with stability guarantees via CLF: it is proven that the robot reaches the desired trajectory even with unknown parameters (payload mass, friction).
·Biology. Stability analysis of population models (Lotka–Volterra, epidemiological models): a Lyapunov function shows which equilibrium (survival/extinction) the system converges to.
·Machine learning. Analysis of gradient descent convergence: the loss function L(θ) is a Lyapunov function, L̇ = −||∇L||² ≤ 0 for GD with a small step size.

Linear systems are stable if all eigenvalues of the matrix A lie in the left half-plane—a simple and convenient criterion. But real systems are often nonlinear: a pendulum, a chemical reactor, an electrical grid, a neural network. In 1892, Aleksandr Lyapunov proposed a universal method for analyz...

System: ẋ = f(x), f(0) = 0 (the origin is an equilibrium point). Without loss of generality, we consider the stability of zero—other equilibria are reduced to zero by a change of variables.

Lyapunov stability. The equilibrium x = 0 is stable if for every ε > 0 there exists δ > 0 such that ||x(0)|| < δ implies ||x(t)|| < ε for all t ≥ 0. Small perturbations remain small.

Asymptotic stability. Stable AND x(t) → 0 as t → ∞. Perturbations not only remain small, but also die out.

Stochastic Optimal Control

Stochastic systems, the Kalman filter, and stochastic dynamic programming

Stochastic Systems and the Kalman Filter

Linear Stochastic System → Kalman Filter Algorithm → Properties → Numerical Example: Object Tracking → Nonlinear Extensions → Real-world Applications

Formulas

Problem: Based on all measurements y_0, y_1, ..., y_t, reconstruct the best estimate x̂_{t|t} = E[x_t | y_0, ..., y_t].

·x̂_{t|t−1} = A·x̂_{t−1|t−1} + B·u_{t−1} (mean by dynamics).
·P_{t|t−1} = A·P_{t−1|t−1}·Aᵀ + Q (covariance: dynamics + noise).
·Innovation: r_t = y_t − C·x̂_{t|t−1}.
·Kalman gain: K_t = P_{t|t−1}·Cᵀ·(C·P_{t|t−1}·Cᵀ + R)⁻¹.
·Updated estimate: x̂_{t|t} = x̂_{t|t−1} + K_t·r_t.
·Updated covariance: P_{t|t} = (I − K_t·C)·P_{t|t−1}.
·GPS and inertial navigation (INS). Integration of accelerometers (accurate in the short term, drift) and GPS (accurate once per second, no drift) via EKF. Accuracy 1–10 m in a smartphone, < 0.1 m i...
·Apollo (1969) and Space Shuttle. Navigation to the Moon was computed onboard by the Kalman filter. MIT program (Battin), project lead — Kalman personally consulted.
·Finance. Estimation of hidden volatility (stochastic volatility models) — nonlinear Kalman filter. Market-making trading strategies use the KF to estimate fair value.
·Robotics and SLAM. Simultaneous Localization and Mapping: a robot builds a map and simultaneously localizes itself. EKF-SLAM, GraphSLAM — core of autonomous Roomba vacuum cleaners and Amazon Kiva w...
·Biomedicine. Wearable ECG monitors with motion artifact filtering. Estimation of blood oxygen saturation in pulse oximeters.

Real-world systems are subject to random disturbances (gusts of wind, thermal noise, unaccounted dynamics), and measurements are affected by sensor noise. Deterministic models and Luenberger observers are insufficient here: it is necessary to explicitly account for the probabilistic nature of unc...

Discrete model: x_{t+1} = A·x_t + B·u_t + w_t (system noise), y_t = C·x_t + v_t (measurement noise).

Noises: w_t ~ N(0, Q), v_t ~ N(0, R), independent from each other and from x_t. Q (n×n), R (p×p) are covariance matrices.

Problem: Based on all measurements y_0, y_1, ..., y_t, reconstruct the best estimate x̂_{t|t} = E[x_t | y_0, ..., y_t].

Stochastic DP and LQG Control

Stochastic Bellman Equation → Numerical Example: Inventory Management Problem → Separation Principle in LQG → Stochastic MPC (Model Predictive Control) → Numerical Example: LQG for a Double Integrator → Reinforcement Learning (RL) — Modern Continuation → Real Applications

Formulas

LQG problem: min E[Σ_{t=0}^T (x_tᵀ·Q·x_t + u_tᵀ·R·u_t)] for a stochastic linear system:Separation theorem (Joseph-Tou, 1961): Optimal control u_t* = −K·x̂_{t|t}, where:

·K — LQR gain matrix from the deterministic problem (solution of algebraic Riccati with A, B, Q, R).
·x̂_{t|t} — state estimate from the Kalman filter with matrices A, C, W, V.
·Naturally accounts for constraints on u and x.
·Adaptive: recalculates the plan when conditions change.
·Applicable to nonlinear systems.
·Scenario approach: generate M = 100 scenarios w^{(i)}_{0:N}, solve the problem for each, select the robust one.
·Tube MPC: design a "tube" of allowable trajectories, guaranteeing constraint satisfaction for all allowable w.
·Chance-constrained MPC: P(C·x_t ≤ d) ≥ 1 − ε — probabilistic constraints.
·Q-learning: Q(x, u) ← Q(x, u) + α·[r + β·max_{u'} Q(x', u') − Q(x, u)].
·DQN (Mnih 2015): Q is approximated by a neural network, trained on batches from a replay buffer.
·Actor-Critic, PPO, SAC: train policy and V/Q in parallel.
·HVAC and smart buildings. Stochastic MPC balances weather forecasts, electricity prices, and comfort. Achieves 15–30% energy savings.
·Portfolio management. Dynamic asset allocation taking into account stochastic returns — an LQG-type problem in linearization; in nonlinear form — stochastic control with HJB.
·Process control (refining, chemistry). MPC is the industrial standard: Honeywell APC systems, AspenTech DMCplus, ExxonMobil PIC. Thousands of installations.
·Autonomous vehicles. Tesla Autopilot, Waymo use MPC in real time for maneuver planning. Accounting for uncertainty in movements of other cars — stochastic MPC.
·Energy. Optimal control of a power station with renewables: stochastic demand, prices, solar/wind generation — stochastic DP problem.

When a system is random, the optimal control itself becomes a "function of the stochastic history." Stochastic DP generalizes the Bellman equation to the case of random transitions and discounted expected rewards — this is the mathematical foundation of reinforcement learning (RL). A special case...

Problem: max E[Σ_{t=0}^T β^t·r(x_t, u_t)] subject to x_{t+1} = f(x_t, u_t, w_t), w_t ~ p(w).

Stochastic Bellman equation: V_t(x) = max_{u ∈ U} [r(x, u) + β·E_{w}[V_{t+1}(f(x, u, w))]].

Backward induction: V_T(x) = r_T(x) (terminal). For t = T−1, ..., 0: V_t(x) = max_u [r(x, u) + β·Σ_{x'} P(x' | x, u)·V_{t+1}(x')].

Reliability of Control and Robust Control

Sources of Uncertainty → Measures of Robustness → $H_\infty$ Control → $\mu$-Synthesis → Adaptive Control → Numerical Example: Pendulum with Unknown Length → Real Applications

·Aviation. All passenger aircraft Airbus A320–A380 are designed on principles of $H_\infty$/$\mu$-synthesis, including “control law switching” system in failures. F-16, F-22—gain-scheduled controlle...
·Hard disks and drives. Head positioning control onto track—a high-speed $H_\infty$ regulator considering flexibility of suspension. Precision 1–10 nm.
·Nuclear reactors. Power control with stability guarantees under temperature changes, fuel burnup—$H_\infty$ synthesis.
·Semiconductor manufacturing. ASML EUV lithography machines—positioning control with accuracy < 1 nm at 10g accelerations. Robust synthesis—a critical component.
·Automotive cruise control. Adaptive controllers (ACC) account for changes in mass (cargo, passengers) and road slope.

LQR/LQG assume that we know the exact model of the system. But real parameters (mass, stiffness, resistance) are known only approximately: manufacturing tolerances, wear, temperature fluctuations. Moreover, real systems have unmodeled dynamics (for example, the flexibility of metal, which is igno...

Parametric. Specific model parameters are inaccurate: mass $m \in [0.9, 1.1]$ kg, spring stiffness $k \in [k_0(1 − 0.2), k_0(1 + 0.2)]$. Described by intervals or probabilistically.

Structural (unmodeled dynamics). Actual transfer function $G_\text{real}(s) = G_\text{nominal}(s) \cdot (1 + \Delta(s))$ where $\Delta$ is an unknown “perturbation”, usually bounded by the norm $||\Delta||_\infty < \gamma$. For example, flexible modes of the structure, which manifest at high fr...

Disturbances and noises. External forces (wind gusts, road bumps), sensor noises. Described as stochastic processes or bounded signals ($||d||_2 \leq D$).