Dynamic Programming in Economics

Dynamic programming (DP) is a method for solving intertemporal optimization problems based on Bellman's principle of optimality: the optimal strategy on any subpath of an optimal trajectory is itself optimal. In economics, DP is used for the analysis of growth, savings, firm management, and asset pricing. Modern macroeconomics (RBC, DSGE, Bewley models) is entirely built on DP methods.

Ramsey's Optimal Growth Problem

Continuous-time version (1928):

$ \max_{c(t)} \int_0^\infty e^{-\rho t} u(c(t)),dt \quad \text{subject to} \quad \dot{k} = f(k) - c - \delta k, \quad k(0)=k_0, \quad c \geq 0 $

Explanation of variables and parameters:

• $k(t)$ — capital stock per capita
• $c(t)$ — consumption per capita
• $\rho$ — discount rate (impatience of the consumer)
• $\delta$ — capital depreciation rate
• $f(k)$ — production function (Cobb–Douglas: $f = k^\alpha$)
• Constraint $\dot{k}=f(k)-c-\delta k$: capital change = production − consumption − depreciation

Pontryagin's Maximum Principle: Hamiltonian $H = u(c) + \mu[f(k) - c - \delta k]$. Optimality conditions: $\partial H/\partial c = 0 \to u'(c) = \mu$ (marginal utility of consumption = price of capital). $\dot{\mu} = \rho\mu - \partial H/\partial k \to \dot{\mu} = (\rho + \delta - f'(k))\mu$.

Euler equation for consumption:

$ \frac{\dot{c}}{c} = \frac{f'(k) - \delta - \rho}{\sigma(c)} $

Here $\sigma(c) = -c u''(c)/u'(c)$ is the intertemporal elasticity of substitution (IES). When $u = \ln c$: $\sigma = 1$, so $\dot{c}/c = f'(k) - \delta - \rho$. Intuition: consumption grows if the return on capital $f'(k)$ exceeds $\delta + \rho$ (returns exceed depreciation plus impatience).

Bellman Equation

Discrete recursive problem:

$ V(x) = \max_{x' \in \Gamma(x)} [u(x,x') + \beta V(x')] $

$x$ — state (capital, wealth), $x'$ — next state, $\beta \in (0,1)$ — discount factor, $\Gamma(x)$ — admissible set, $u$ — current reward.

Intuition: the value of state $x$ = best current payoff $+$ discounted value of the future optimal state. Recursion allows “breaking up” an infinite horizon into one-step problems.

Bellman Fixed Point Theorem: The operator $T(V)(x) = \max_{x' \in \Gamma(x)} [u(x,x') + \beta V(x')]$ is a contraction with coefficient $\beta \in (0,1)$ in the metric space of bounded functions (sup-norm). Corollary: there exists a unique fixed point $V^$. Iterations $V_{n+1} = T(V_n)$ converge: $|V_n - V^| \leq \frac{\beta^n}{1-\beta} \cdot |V_1 - V_0|$.

Optimal policy: $g(x) = \arg\max_{x' \in \Gamma(x)} [u(x,x') + \beta V^*(x')]$ — policy function, which determines optimal behavior in every state.

Steady State and the “Golden Rule”

Steady state: $\dot{k} = 0$, $\dot{c} = 0$. From Euler's equation: $f'(k^*) = \rho + \delta$ — “modified golden rule” (with impatience).

Pure Phelps’ Golden Rule (1961): $\max c = f(k) - \delta k \to f'(k_{GR}) = \delta$. At $\rho > 0$: $k^* < k_{GR}$ — impatience lowers capital accumulation below the “golden” level.

Transitional dynamics: if $k_0 < k^*$, then $k$ grows, $c$ grows, so the system moves toward the steady state along the “saddle path”. Uniqueness: only one trajectory $c(t)$ leads to the steady state (transversality condition).

Numerical methods: Value function iteration (VFI): for the grid $K = {k_1, ..., k_n}$: $V_{j+1}(k_i) = \max_{k' \in K} [u(f(k_i) - k') + \beta V_j(k')]$. Projection method with Chebyshev polynomials: approximate $V^$ by a sum of Chebyshev polynomials. Perturbation method: log-linearization around $k^$.

Numerical Example

$u(c) = \ln(c)$, $f(k) = k^{0.36}$, $\beta = 0.95$, $\delta = 0.10$. Trial form: $V^(k) = A + B\cdot \ln(k)$. From the Bellman equation: $A + B \cdot \ln(k) = \max_c [\ln(c) + \beta(A + B\cdot \ln(k^{0.36} - c))]$. First order condition (FOC): $1/c = \beta B (-1)/(k^{0.36} - c)$. Solution: $c^ = k^{0.36}/(1 + \beta B)$. Substituting into the Bellman equation: $B = \alpha \beta B/(1 - \beta(1-\alpha)) \to B = \alpha/(1 - \alpha\beta)$. At $\alpha = 0.36$, $\beta = 0.95$: $B = 0.36/(1 - 0.36\cdot 0.95) \approx 0.36/0.658 \approx 0.547$. Policy function: $c^* = k^{0.36}(1-\alpha\beta) = k^{0.36} \cdot 0.658$.

At $k_0 = 10$: $k^{0.36} \approx 3.17$, $c^* = 0.658\cdot 3.17 \approx 2.09$, $k_1 = 3.17 - 2.09 + (1-0.1)\cdot 10 = ...$ and we iterate further.

Exercise: $u(c) = \ln(c)$, $f(k) = k^{0.36}$, $\beta = 0.95$, $\delta = 0.10$. Numerically solve the VFI problem on a grid $k \in [0.01, 3]$ with $N = 200$ points. (1) Plot $V^(k)$ and compare it to the analytical $A + B\cdot \ln(k)$. (2) Plot the policy function $c^(k)$. (3) Simulate 100 steps with $k_0 = 0.5$. (4) Find the steady-state $k^, c^$ analytically and compare to the simulation.