Companion guide / Decision Science
The State Variable
What your decision system knows, and what it costs when it is wrong
Physical, information, and belief states; what belongs in the state and what does not; and what a late, noisy, biased, or incomplete state costs each policy class.
Adam DeJans Jr. | | A companion to The Sequential Decision Toolbelt
The argument in brief
A policy can be no better than the state it reads.
Every policy in The Sequential Decision Toolbelt has the same form, . Teams spend most of their effort on and : which class, which solver, which tuning method. The argument of this guide is that deserves at least as much attention, for three reasons.
First, the state defines what the policy can know. A rule, an optimizer, or a learned value function cannot respond to a commitment, a pipeline order, or a seasonal peak that is not in its state. Second, the state decides which policy classes are feasible at all, because some classes need the state to be small and others need it only as a starting point. Third, the state is where production systems quietly fail: data arrive late, counts drift from the shelf, and fields that the model needs are missing. None of these failures produce an error message; they produce plausible decisions that cost money.
The experiment in Part 6 puts numbers on the third point, using the Line 3 production line from the toolbelt. An inventory feed that arrived one week late raised the weekly cost of every tuned policy by between 26% and 37%, and the most sophisticated policies lost the most. Re-tuning the policies on the late data recovered little. Projecting the late snapshot forward with the production the policy had itself released, a rudimentary belief state, recovered 92% of the loss on average. Meanwhile, random count errors of about 20% and a consistent 20% overstatement of stock cost almost nothing. On this line, the timeliness and completeness of the state mattered far more than its precision, and the remedy was a better state, not a better policy.
Part 1
What a state is
Not a node in a network, not a snapshot of the database, and not whatever the planning system happens to store.
Powell's definition is deliberately operational. The state at time is the information, drawn from everything that has happened so far, that is necessary and sufficient to compute three things from that point on: the cost or contribution of a decision, the decision itself under the chosen policy, and the transition to the next state given new information (Powell, state variables). Two consequences follow immediately. The state is defined relative to the model and the policy, so changing the policy can change what must be in the state. And a system modeled this way is Markov by construction: anything from the past that matters is, by definition, carried in the state.
Powell divides the state into three components that may appear alone or together.
Physical (resource) state
Quantities of things you control or consume: inventory, machines, people, vehicles, money.
Line 3
End-of-week inventory of families A, B, and C.
A real plant
On-hand by location and lot, work in process by routing step, open purchase and production orders in the pipeline, machine status and hours since maintenance, last product run on each line, crew roster.
Information state
Other known values and parameters the decision or the costs depend on: prices, calendars, forecasts, rules in force.
Line 3
The week of the season, which fixes the demand forecast.
A real plant
Forecast distributions by item and week, confirmed customer orders and due dates, supplier lead-time quotes, the shift calendar, current prices and penalties, frozen-window boundaries.
Belief state
What you know about quantities you cannot observe perfectly, held as a distribution or its statistics.
Line 3
Absent in the base model, where inventory and the demand distribution are known exactly. Present once counts are noisy.
A real plant
Record accuracy by location, the estimated distribution of supplier reliability, a machine's estimated wear, the uncertainty of the forecast itself, a learned yield curve and its confidence.
Powell also warns against two habits that make states too small. The first is to think of a state as a position, a node in a network or a square on a board, when the decision also depends on how you got there; his example is a path problem in which the state must carry the costs already incurred on the path, not just the current node. The second is to treat lagged values as history rather than state: if a price model uses the last three prices, then all three are part of the state. The practical rule is that "history" is not a separate category. Either a piece of the past is needed for the decision, the cost, or the transition, in which case it belongs in the state, or it is not, in which case it can be dropped.
Part 2
Two tests for a state
A state is complete when it passes both. Most incomplete states fail the second.
Test 1 · Decision sufficiency
If two situations share the same state, should the decision be the same?
If an experienced planner would act differently in two situations your state cannot tell apart, something the decision depends on is missing. Planner overrides are the best source of such cases (see Planner overrides are training data).
Test 2 · Transition sufficiency
Can you compute next period's state and costs from this state and the new information alone?
If the simulator needs to look up anything from the past to advance one period, that thing belongs in the state. This test catches pipeline orders, lot ages, and sequence-dependent setups that a decision-only review misses.
Turning history into state
Most real states are larger than the first draft because some of the past matters. The table lists recurring cases in manufacturing and supply chain, the history that matters, and the state variable that carries it forward.
| Situation | The past that matters | State variable that carries it |
|---|---|---|
| Replenishment with lead times | Orders placed but not yet received | Pipeline by arrival date, not a single "inventory position" number |
| Sequence-dependent changeovers | Which product ran last on each line | Current setup state per line |
| Perishable or shelf-life items | When each unit was made | Inventory by age bucket or expiry date |
| Backorders and due dates | Which orders are late and by how much | Backlog by due date and customer priority |
| Condition-based maintenance | Run hours, cycles, and readings since the last intervention | Hours since service plus the latest condition estimate |
| Forecast-driven planning | Recent demand and the forecast's recent errors | The current forecast distribution, including its spread |
| Contracts with cumulative terms | Volume already bought against a minimum or a tier | Cumulative volume to date per contract |
| Supplier learning | Each supplier's delivery record | Estimated reliability and its uncertainty (a belief state) |
The first row deserves its own article, and has one: Inventory position is a state, not a number. Summing on-hand and on-order into one figure discards when the orders arrive, which is precisely what a decision facing a stockout next week needs to know.
Part 3
Before and after the decision
The post-decision state is the quiet idea that makes value function approximation practical.
Between two pre-decision states there is a useful intermediate point: the state immediately after the decision is made and before new information arrives. Powell calls it the post-decision state, . The sequence of a period then reads
On Line 3 the pre-decision state is the inventory left at the end of the week; the post-decision state is that inventory plus what the planner has just released to the line, , the units that will be available when next week's demand arrives. The post-decision state is deterministic given the decision, which is why it matters: a value function written on can be placed directly inside an optimization model, because choosing chooses without any expectation inside the optimization. The VFA in the toolbelt's worked example is built exactly this way (see the VFA class profile).
For practitioners the post-decision state is also a useful auditing device. It is the state the business is committed to before the world responds, and it is what a planning meeting is actually choosing.
Part 4
The state decides which policies are feasible
Each policy class asks something different of the state. That, more than taste, often settles the architecture.
PFA Policy function approximation
Needs from the state: A few features of the state that the rule can read directly.
A PFA scales to any state size as long as the decision depends on a small summary: inventory position, a due date, a health score. A lookup table over the full state does not scale at all.
State risk: Information outside the chosen features is invisible. The rule cannot respond to a commitment it does not read.
CFA Cost function approximation
Needs from the state: The current state as the data of an optimization model.
Scales well: a large state simply becomes a large model instance. The tuned penalties stand in for the future, not for the state.
State risk: Treats the state as exact. Errors in the data flow straight into the model's constraints.
VFA Value function approximation
Needs from the state: A state compact enough to approximate a value function over it.
The most demanding class. Exact dynamic programming enumerates states; even Line 3, with three families and 13 season phases, has about 45 million states if inventories range up to 150 units. Approximations must exploit structure such as separability.
State risk: A value learned over states that rarely occur in training is unreliable exactly where it is needed.
DLA Direct lookahead approximation
Needs from the state: The current state as the initial condition of a lookahead model, plus forecasts as the information state.
Scales with the size of the lookahead model, not with the number of possible states, which is why DLAs dominate large industrial problems.
State risk: Forecasts become part of the state. A lookahead fed a stale or flattened forecast plans confidently for the wrong future.
A plant with 200 items and a hundred plausible inventory levels for each has on the order of inventory states before machines, orders, or forecasts are counted. No value function or lookup table represents that directly. This is why the curse of dimensionality is first a property of the state and only second a property of the algorithm, and why the toolbelt's advisor asks about the state before it asks about anything computational (see the advisor).
Part 5
What to leave out
Every variable in the state is a commitment to measure it, keep it current, and trust it. Completeness has a price.
The two tests in Part 2 say what must be in the state for the model to be correct. In practice the state used by a deployed policy is always smaller than the true state, for good reasons, and the craft lies in choosing what to leave out.
Leave out what does not change the decision. A variable that fails to move any decision across the range of states you will encounter adds cost and no value. The value of information is a property of the decision, not of the data (see The value of information depends on the decision).
Aggregate where the decision is made at a coarser grain. If production is planned by family, the state can carry inventory by family, provided the aggregation does not hide a constraint that binds at the item level (see The aggregation level is a modeling decision and Forecast at the grain of the decision).
Prefer a variable you can keep accurate over a richer one you cannot. A state element that arrives late or wrong can be worse than none, because the policy acts on it with full confidence. The experiment in Part 6 measures how much worse.
Leave out what the policy cannot use, but say so. A deterministic lookahead has no use for the spread of the forecast; a tuned buffer stands in for it. That is a legitimate simplification as long as it is written down, because the day someone adds a stochastic lookahead, the spread must come back into the state.
Part 6
When the state is wrong: an experiment
Same plant, same policies, same demand. The only change is what the policy is allowed to see.
We return to Line 3 from the toolbelt: three product families on one line, changeovers, a seasonal peak, and lost sales (see the worked example). Six of the policies built there, each at the parameters tuned with an accurate state, now decide from four kinds of degraded state while the true inventory drives the simulation and the costs. The degradations are the ones operations teams meet in practice: latency, where the inventory feed lags one planning cycle; noise, where each family's record is off by about 20% at random; bias, where records consistently overstate the shelf by 20%, as phantom inventory does; and an incomplete information state, where inventory is accurate but the seasonal forecast is missing. Every policy faces the same 100 held-out one-year demand scenarios, so each cell is a paired comparison with the same policy on an accurate state.
Exhibit
What a wrong state costs each policy
Increase in average weekly cost against the same policy deciding from an accurate state, with 95% confidence intervals
| Policy | Accurate state cost / week | One week late | Late, projected | Noisy counts | Phantom inventory | Season missing |
|---|---|---|---|---|---|---|
| Hand-set rule PFA | 383.6 | +39.4 ± 2.1 · +10.3% | +3.4 ± 1.6 · +0.9% | −0.4 ± 1.2 · −0.1% | +2.4 ± 0.8 · +0.6% | +0.0 ± 0.0 · +0.0% |
| Tuned PFA PFA | 288.4 | +90.2 ± 3.3 · +31.3% | +3.5 ± 2.6 · +1.2% | −0.1 ± 2.6 · −0.0% | −2.4 ± 2.4 · −0.8% | +9.5 ± 2.9 · +3.3% |
| Tuned CFA CFA | 277.3 | +73.2 ± 2.5 · +26.4% | +6.2 ± 1.9 · +2.2% | +4.0 ± 2.1 · +1.4% | −3.2 ± 2.1 · −1.2% | +18.7 ± 2.5 · +6.8% |
| Buffered DLA CFADLA | 273.4 | +85.7 ± 2.9 · +31.4% | +9.6 ± 2.0 · +3.5% | +3.7 ± 1.8 · +1.4% | −0.3 ± 1.7 · −0.1% | +22.3 ± 2.1 · +8.1% |
| VFA (trained by ADP) VFA | 271.5 | +89.3 ± 2.2 · +32.9% | +6.4 ± 1.2 · +2.4% | +3.6 ± 0.8 · +1.3% | +0.1 ± 0.8 · +0.0% | +0.9 ± 0.6 · +0.3% |
| DLA + VFA DLAVFA | 268.9 | +99.2 ± 2.5 · +36.9% | +8.3 ± 1.3 · +3.1% | +3.6 ± 1.2 · +1.3% | +3.0 ± 1.1 · +1.1% | +26.9 ± 2.3 · +10.0% |
- One week late.
- The policy sees last week's inventory, as if the data feed lagged one planning cycle and nothing corrected it.
- Late, projected.
- The same late snapshot, projected forward with the production the policy itself released and the expected demand: a simple belief state.
- Noisy counts.
- Each family's recorded inventory is off by about 20% at random, unbiased on average.
- Phantom inventory.
- Records overstate what is on the shelf by 20%, consistently.
- Season missing.
- Inventory is accurate, but the information state lacks the seasonal forecast: the policy plans to the annual average.
Computed by scripts/state-sim.mjs (source hash 68c75f223e8abd74) on the Line 3 model, 100 held-out one-year scenarios, common random numbers. Every policy keeps the parameters tuned with an accurate state. The true inventory drives the simulation and the costs; only what the policy sees changes. Illustrative for this model only.
Four findings stand out. Each is a statement about this model; the mechanisms behind them are general, the magnitudes are not.
1. Lateness was the expensive error, and it hurt the best policies most. With an uncorrected one-week lag, the tuned policies lost between 26.4% and 36.9% of their accurate-state cost; the DLA + VFA hybrid, the best policy with an accurate state, lost 36.9%. The hand-set rule lost only 10.3%, because it reacts to inventory sluggishly to begin with. The mechanism is visible in the model: a snapshot taken before last week's production arrived does not contain that production, so a responsive policy sees a shortfall that no longer exists and tends to fill it a second time. The more precisely a policy responds to its state, the more precisely it responds to a wrong one.
2. A better state fixed most of it. The "Late, projected" column feeds the policy the same late snapshot, adjusted by two things the planner already knows: the production the policy released last week and the expected demand since. Nothing else changes. Losses fall to between 1.2% and 3.5%, recovering 92% of the lateness loss on average. This is a belief state in miniature: the best estimate of the current state given late data, built from the transition model and the policy's own decisions. It costs one line of code.
3. Random noise and consistent bias were cheap here. No policy's cost moved by more than 1.4% under noisy counts or 1.2% under a 20% overstatement, and some moved slightly down. The tuned policies carry generous cover, and a proportional error shifts their effective targets by roughly as much as the coarseness of their tuning. This result should not be generalized. Under lean inventories, or with phantom stock that persists on locations whose true inventory is zero, the same errors cause stockouts that the records conceal, which is how phantom inventory does its damage in retail.
4. A missing forecast hurt the lookaheads most. Without the season in the information state, the buffered DLA lost 8.1%, the DLA + VFA hybrid 10.0%, the CFA 6.8%, and the tuned PFA 3.3%; the hand-set rule, which never used the season, lost nothing. The VFA lost only 0.3%, and the reason is instructive rather than reassuring: its learned value function is indexed by week of season, so it carries the seasonal pattern in its parameters even when the forecast is removed. Information can migrate from the state into a policy's parameters, where it is harder to see, audit, or update when the season changes.
Exhibit
Re-tuning on the state you will actually have
Loss against an accurate state, before and after re-tuning each policy's parameters on the degraded state
Tuned PFA
Tuned CFA
Buffered DLA
Re-tuned by the same grid searches as the original tuning, on 40 training scenarios with the degraded state; evaluated on the 100 test scenarios. A negative "after" value would mean the re-tuned policy beats the original one, which can happen only through noise or a better parameter set. Illustrative for this model only.
Re-tuning is the wrong tool for a missing state. The natural response to degraded data is to re-tune the policy on the data it will actually receive, and Part 7 of the toolbelt recommends exactly that. It helps when the error is a matter of calibration. It does little when the error removes information. Against the uncorrected lag, the best re-tuning result came from the Tuned PFA, which recovered 17% of its loss; the other policies recovered little or none, and in a few cells the re-tuned policy did slightly worse on the test scenarios than the original, which is what selecting the best of many noisy training results looks like out of sample. Tuning changes how a policy responds to its state. It cannot restore what the state no longer contains. The projection in finding 2, which changes the state rather than the policy, recovered far more.
The practical sequence follows directly. First make the state as timely and complete as the decision needs, using the transition model and the policy's own decisions to project late data forward. Then tune the policy on that realistic state. Only then compare policy classes.
Part 7
Belief states in practice
When you cannot observe the state, the state becomes what you believe about it.
Powell's answer to missing information is direct: replace it with a probabilistic belief, and make the belief part of the state. In the noisy-count experiment, the true inventory is unknown, and the honest state is not the recorded number but a distribution around it. Practitioners rarely build formal belief states, yet they use three practical forms of the idea constantly.
- Carry the uncertainty explicitly. Record accuracy by location, forecast error by item, and lead-time variability by supplier are belief-state statistics. A policy that knows a location's counts are unreliable can hold more protection there, exactly as a tuned buffer does on average.
- Update the belief with every observation. A sale that fails because the shelf was empty, a pick that finds fewer units than recorded, and a delivery that arrives late are all observations about the hidden state. Bayesian updating, or in continuous settings a Kalman filter, turns them into better estimates instead of discarding them (see Lost sales are silent demand).
- Treat measurement as a decision. A cycle count, an inspection, or a sensor upgrade buys information. Its value is the improvement in decisions it produces, which the experiment above estimates for Line 3: the loss from noisy counts is an upper bound on what perfect counts are worth to each policy, per week.
The third point turns data quality from an IT concern into an economic one. A team deciding whether to invest in real-time inventory visibility can estimate its value in exactly this way: simulate the policy it actually runs, on the state it will actually have, with and without the investment.
Part 8
A state audit
A one-page register that every production decision system should have, and few do.
For each element of the state, the register records where it comes from, how late it is when the decision is made, how accurate it is, who owns it, and what the policy does when it is missing. The example below is for a Line 3 at plant scale. The point is not the format but the discipline: an element nobody owns will eventually be wrong, and an element with no fallback will eventually stop the decision.
| State element | Component | Source | Latency at decision | Accuracy | Owner | If missing |
|---|---|---|---|---|---|---|
| On-hand by family and location | Physical | WMS | Nightly snapshot, up to 24 h | Cycle-count accuracy 96% | Warehouse lead | Use last snapshot plus booked movements; widen buffer |
| Released production not yet complete | Physical | MES | Real time | High | Production control | Assume released quantities complete on schedule |
| Current setup on each line | Physical | MES | Real time | High | Line supervisor | Assume worst-case changeover |
| Demand forecast by family and week | Information | Forecasting system | Weekly, Sunday | Tracked error by horizon | Demand planning | Fall back to seasonal profile |
| Confirmed orders and due dates | Information | ERP | Hourly | High | Customer service | Plan to forecast only; flag |
| Record accuracy by location | Belief | Cycle-count history | Monthly | Estimated | Inventory control | Use plant-wide average |
Illustrative register; the systems, latencies, and accuracies are examples, not measurements.
Six questions to ask of any state
- Does it pass both tests in Part 2, checked against real planner overrides and against the simulator's transition code?
- Which elements are physical, which informational, and which are beliefs that ought to carry their own uncertainty?
- For each element, how late is it at the moment the decision is made, and how wrong is it typically?
- Has the policy been tuned on the state it will actually receive, with that latency and that error, rather than on clean historical data?
- What is the measured cost, per period, of each element's latency and error, and therefore what is better data worth?
- What does the policy do when an element is missing, and has that fallback been simulated?
Sources and further reading
Primary sources, credits, and related reading
The definition of the state, its three components, the post-decision state, and the treatment of missing information as belief are due to Warren B. Powell. The two tests, the history-to-state table, the class requirements, the state audit, and the Line 3 experiment are original to Bit Bros.
- State variables, warrenpowell.org.
- The universal modeling framework, warrenpowell.org.
- Powell, W. B. (2022). Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions. Wiley. Chapter 9 treats the modeling of state variables.
- Powell, W. B. (2011). Approximate Dynamic Programming: Solving the Curses of Dimensionality (2nd ed.). Wiley. The post-decision state is developed here.
Related Bit Bros reading
Flagship guide
The Sequential Decision Toolbelt
Physical state
Inventory position is a state, not a number
Latency
Decision latency is a cost
Latency
Decision latency is a model input
Belief state
The value of information depends on the decision
Data
Data access is a decision constraint
Information state
Lead time is a distribution, not a number
Observation
Sales are not demand
Information state
Forecast vintage leakage
Work with Bit Bros
Need help designing a decision system for your operations? Talk to Bit Bros.
We start where most projects should: with what the decision actually knows, how late and how wrong it is, and what better data would be worth before anyone buys it.