Dynamic Decision Making (Finite Horizon Bellman)

At each step you choose among actions affecting future states—classic MDP-style thinking without full RL: Bellman idea in discrete small graphs.

Why this shows up in the real world

Inventory restocking under uncertain demand (sometimes approximated deterministically). Game AI lookahead.

Core idea (explained for students)

Write V[t][state] = max_a reward + V[t+1][next(state,a)] backwards in time for finite horizon.

Try this in Python

def finite_horizon_value(states: int, T: int, rew, trans) -> float:
    V = [0.0] * states
    for _ in range(T):
        newV = [0.0] * states
        for s in range(states):
            best = -10**18
            for a in range(2):
                ns = trans[s][a]
                best = max(best, rew[s][a] + V[ns])
            newV[s] = best
        V = newV
    return max(V)


trans = [[1, 0], [0, 1]]
rew = [[1, 0], [0, 2]]
print(round(finite_horizon_value(2, 5, rew, trans), 3))

Common mistakes

• Infinite horizon without discount—values may diverge.
• Stochastic transitions need expectation.

Key takeaways

• Discount factor γ<1 stabilizes many textbook models.