Problems with Finite and Infinite Horizon

Engineering problems (rocket landing, satellite maneuvering) usually have a finite horizon — the criterion is tied to the end moment. Economic problems (consumption, growth) are often infinite — we want to optimize welfare “forever.” These two classes require different transversality conditions and lead to different numerical methods. A special case is linear-quadratic problems (LQR), which allow an explicit solution via the Riccati equation.

Finite Horizon Problem

The transversality condition depends on how the boundary conditions are set at moment T.

Case 1: x(0) and x(T) are fixed. The conjugate variables ψ(0), ψ(T) are determined from the solution of the boundary value problem (x, ψ).

Case 2: x(T) is free, T is fixed. Additional condition: ψ(T) = ∂φ/∂x(x(T), T). If there is no terminal penalty (φ = 0), then ψ(T) = 0.

Case 3: T is also free (minimum-time or variable-horizon problem). Additionally: H(x(T), u(T), ψ(T), T) = 0. This explains why in the min T problem the Hamiltonian at the terminal moment equals zero.

Legendre–Clebsch condition. For sufficiency of minimum: the matrix ∂²H/∂u² must be positive semi-definite at all trajectory points. If it is strictly positive — we have a “regular” interior optimum; if zero or negative — more subtle conditions are needed (e.g., the Gilbert–Blatt condition).

Infinite Horizon Problem

Standard form: max ∫₀^∞ e^{−ρt}·F(x, u) dt subject to ẋ = f(x, u), x(0) = x₀.

Discounting e^{−ρt} with ρ > 0 ensures convergence of the integral (if F is bounded). Without discounting, one has to use criteria such as catching-up or overtaking.

Transversality condition: lim_{T→∞} e^{−ρT}·ψ(T)·x(T) = 0. Economically: “the shadow value of capital at infinity is zero” — otherwise, one could increase utility by increasing (or decreasing) the capital stock.

Current-value Hamiltonian: It is convenient to denote m(t) = e^{ρt}·ψ(t) — the current-value of the shadow price. Then the equation for m: ṁ = ρm − ∂Ĥ/∂x, where Ĥ = e^{ρt}·H is the current-value Hamiltonian.

Stationary optimal state. If f, F do not explicitly depend on t, look for (x*, u*) with f(x*, u*) = 0, ∂Ĥ/∂u = 0, ρ·m = ∂Ĥ/∂x. This yields the “golden rule” — the long-run equilibrium of the model.

Linear-Quadratic Problems (LQR)

This is a foundational class of problems — the only one where the optimal control is explicitly expressed in feedback form.

LQR formulation: min ∫₀^∞ (xᵀ·Q·x + uᵀ·R·u) dt subject to ẋ = A·x + B·u.

Weights: Q ≥ 0 — penalty for deviation of state from zero, R > 0 — penalty for “expensive” control. Large Q — we seek to quickly return the system to zero; large R — we economize on control.

Solution via the algebraic Riccati equation (ARE): Optimal u*(t) = −K·x(t), where K = R⁻¹·Bᵀ·P, and P is a symmetric positive definite matrix satisfying:

Aᵀ·P + P·A − P·B·R⁻¹·Bᵀ·P + Q = 0.

The closed-loop system: ẋ = (A − BK)·x is stable (all eigenvalues in the left half-plane) under qualified (A, B), (A, √Q): controllability/observability conditions.

Numerical Example: LQR for a Scalar System

System: ẋ = x + u (unstable), Q = 1, R = 1. ARE: 1·P + P·1 − P·1·1·P + 1 = 0 → 2P − P² + 1 = 0 → P² − 2P − 1 = 0 → P = 1 + √2 ≈ 2.414. K = P ≈ 2.414. Closed-loop: ẋ = (1 − 2.414)·x = −1.414·x — stable with decay rate 1.414. The root of the characteristic polynomial is further to the left than without control.

LQR for the Inverted Pendulum

Linearization near the upright position: A = [0, 1; g/l, 0], B = [0; 1/(m·l²)]. The system is unstable (eigenvalues ±√(g/l)).

Q = diag(10, 1), R = 0.01. Numerically (Python: scipy.linalg.solve_continuous_are): P ≈ [[…]], K ≈ [k₁, k₂]. The closed-loop system is stable, the pendulum is stabilized near vertical when displaced by 5–10°.

Real Applications

• Aircraft autopilot. Each of the loops (pitch, roll, yaw) is designed as LQR around the flight regime. For different speeds/altitudes a set of K is computed, between which “gain scheduling” occurs.
• Segway/two-wheeled robot stabilization. Standard LQR for the model of an inverted pendulum on wheels.
• Power plant control. Maintaining stable frequency (50 Hz) under load fluctuations — LQR control of turbines.
• Finance. The optimal trading strategy with quadratic market impact costs (Almgren–Chriss) reduces to LQR — control = trading speed.

Assignment. LQR for an inverted pendulum on a cart: x = (position, velocity, angle, angular velocity), A, B standard (see textbook). Q = diag(1, 1, 10, 1), R = 0.1. (a) Using scipy.linalg.solve_continuous_are numerically solve the ARE and find K. (b) Check stability: eigenvalues of (A − BK) — all should be in the left half-plane. (c) Simulate with an initial angle deviation of 0.1 rad: plot x(t) for t ∈ [0, 5]. (d) Experiment: reduce Q[2,2] = 1 (smaller angle penalty) — how does the transient response change?