Formulation of the Optimal Control Problem

In the calculus of variations, the sought-for function $y(x)$ is smooth and unconstrained. But imagine a spacecraft with engine thrust $u(t)$: its magnitude cannot exceed the maximum thrust, it can be turned on and off abruptly. Classical calculus is powerless here—new machinery is needed. Optimal control theory, developed by Lev Pontryagin and his school in the 1950s–60s, generalizes the calculus of variations to the case where the “control” $u$ is restricted by a prescribed set, and changes can be discontinuous. This led to one of the greatest achievements in applied mathematics of the 20th century—the Pontryagin Maximum Principle.

Standard Optimal Control Problem

Bolza Problem: $\max J = \varphi(x(T), T) + \int_0^T L(x(t), u(t), t),dt$ subject to:

• State equation: $\dot{x} = f(x(t), u(t), t)$, $x(0) = x_0$.
• Control: $u(t) \in U \subset \mathbb{R}^m$ is the admissible set (often compact).
• Boundary conditions: $\psi(x(T), T) = 0$ (or $x(T)$ is free).

Explanation. $x(t) \in \mathbb{R}^n$ is the state of the system (position, velocity, fuel remaining). $u(t)$ is the control (thrust, rudder angle, tax rate). $L$ is instantaneous utility/cost, $\varphi$ is terminal reward/penalty, $f$ is the law of system evolution.

Special cases.

• Lagrange problem: $\varphi \equiv 0$ (only an integral criterion).
• Mayer problem: $L \equiv 0$ (terminal only).
• Minimum-time problem: $J = T$ (minimize time).

Unlike the calculus of variations, $u(t)$ may be discontinuous and may take values only from $U$. For $U = [u_{\min}, u_{\max}]$ the optimum often lies on the boundary—this gives the characteristic "bang-bang" control.

Pontryagin Maximum Principle

Hamiltonian: $H(x, u, \psi, t) = L(x, u, t) + \psi^\mathsf{T} \cdot f(x, u, t)$.

$\psi \in \mathbb{R}^n$ is the vector of adjoint variables, in meaning— the “shadow price” of the state (cf. Lagrange multipliers: $\psi_i$ shows how much $J$ would increase if one unit were freely added to $x_i$).

PMP (necessary conditions). For the optimal pair $(x^(t), u^(t))$ there exists $\psi(t) \neq 0$ such that:

1. State equation: $\dot{x}^* = \frac{\partial H}{\partial \psi} = f(x^, u^, t)$.
2. Adjoint equation: $\dot{\psi} = -\frac{\partial H}{\partial x}\big|_{x^, u^, \psi}$.
3. Maximum condition: $H(x^(t), u^(t), \psi(t), t) = \max_{u \in U} H(x^*(t), u, \psi(t), t)$ for almost every $t$.
4. Transversality condition: $\psi(T) = \frac{\partial \varphi}{\partial x}(x^*(T), T)$ (if $x(T)$ is free).

The key is point 3: $u^*$ is chosen so as to maximize $H$ pointwise, subject to the restriction $u \in U$. This is stronger than the stationarity condition $\partial H/\partial u = 0$ of classical calculus.

The Difference from Calculus of Variations

If $U = \mathbb{R}^m$ (no constraints) and $H$ is smooth, the max is attained at an interior point: $\partial H/\partial u = 0$. This is equivalent to the Euler–Lagrange equation.

If $U = [a, b]$, the optimal $u^$ may be on the boundary. If $H$ is linear in $u$: $u^ = b$ when the coefficient at $u$ is positive, $u^* = a$ when negative, an indeterminacy on the switching set. This is bang-bang control—abrupt switching between extreme modes.

Example: rocket control. $x = $(position, velocity), control $u =$ thrust $\in [−F_{max}, F_{max}]$. PMP gives $u^* = F_{max}\cdot \mathrm{sign}(\psi_v)$—the engine runs either at full power forward or backward. This explains why a rocket pilot "yanks" the throttle rather than smoothly adjusting thrust.

Numerical Example: Minimum-Time Problem

Transfer $x = (q, v)$ from $(1, 0)$ to $(0, 0)$ in minimal time with $u \in [−1, 1]$: $\dot{q} = v$, $\dot{v} = u$, minimize $T$.

Hamiltonian: $H = -1 + \psi_q \cdot v + \psi_v \cdot u$ (minus 1—since $J = \int 1,dt$ and we maximize $-J$). Adjoint: $\dot{\psi}_q = 0,\ \dot{\psi}_v = −\psi_q$. Hence $\psi_q = \text{const}$, $\psi_v = -\psi_q t + c$—linear in $t$, changes sign at most once. Maximum condition: $u^* = \mathrm{sign}(\psi_v) \in {-1, +1}$. We obtain bang-bang control with a single switch.

Solving numerically: the optimal strategy is to brake $u = -1$ until $v = -1$ (interval $t \in [0, 1]$), then accelerate $u = +1$ until coming to rest at $(0, 0)$ (at $t \in [1, 2]$). Minimum time $T^* = 2$.

Real-World Applications

• Aerospace industry. The Apollo Guidance Computer (1969) used a simplified form of the PMP to optimize the lunar lander trajectory. Modern SpaceX Falcon 9 rockets solve the soft-landing problem in real time via convex optimization based on the PMP.
• Robotics. Minimum-time motion of a manipulator between two positions is an optimal control problem with constraints on torque. Bang-bang control is often optimal but requires smoothing for mechanical stability.
• Autonomous transportation. Tesla Autopilot, Waymo solve trajectory planning tasks taking into account limits on acceleration and turning angle. Modifications of PMP and MPC (Model Predictive Control) are used.
• Epidemiology. The optimal vaccination/lockdown strategy during COVID was modeled as an optimal control problem for the SEIR model with control being the “stringency” of restrictions.

Exercise. Transfer $x = (q, v)$ from $(0, 0)$ to $(1, 0)$ in minimal time with $u \in [−1, 1]$ ($\dot{q} = v,, \dot{v} = u$). (a) Write the Hamiltonian. (b) Derive the form of the optimal $u^(t)$ from the PMP. (c) Show that the optimal strategy is $u = +1$ up to the switching instant $t_s$, then $u = -1$. (d) Find $t_s$ and $T^$. (e) Draw the phase portrait $(q, v)$ with the switching curve $q = \pm v^2/2$.