Demystifying AI Forecasting:
From Black Box to Boardroom Answer

The question nobody prepares for

You ran the Python pipeline. The models ran. The FVA numbers look good. The 12-month forward forecast is sitting in the Excel workbook ready to share. You walk into the executive meeting, present the numbers - and then it happens.

"This says $1.5M for May. How did it come up with that? What went into it?"

If your answer is "the algorithm analyzed the historical data," you have already lost the room. Not because the forecast is wrong - it may be excellent - but because an unexplained forecast is an untrustworthy forecast. The finance team will not load it into the budget. The procurement manager will not size the purchase orders against it. The operating partner will not present it to the board.

This article shares some of the tools we use to answer that question - the feature matrix that shows what the model was actually trained on, the FVA framework for selecting the right model per entity, SHAP values that convert model output into dollar-level attribution per feature, and the forecastability scorecard that determines which entities belong in an automated forecast at all. None of these are exotic. Together they make "how did it come up with that number" a question with a real answer.

What the model actually sees: the training feature matrix

A machine learning demand forecast does not see the business. It sees a table. Each row is one month of data for one grain - a specific combination of facility, end market, and product category. Each column is a feature: a number the model can use to learn the relationship between past conditions and future revenue.

The feature matrix for a typical manufacturing entity contains 30 or more columns, organized into five groups:

• Revenue history (y actuals and lag features) - the target variable y plus lag_1 through lag_12: what revenue was 1 month ago, 2 months ago, and so on back through the training window
• Rolling statistics - 3-month and 6-month rolling mean and standard deviation, capturing recent trend and volatility
• Calendar features - month of year, quarter, year trend, and year-over-year growth rate, giving the model awareness of seasonality and direction
• Backlog lag features - open order balance 1, 2, 3, and 6 months prior, the most direct leading indicator of near-term revenue
• Quote activity features - aggregate new-quote value lagged 1, 2, 3, 6, and 12 months, plus a normalized quote index (current month quotes divided by trailing 12-month average) so the model reads whether the pipeline is running hot or cold relative to its own history

The final row of the training matrix is the prediction row. All feature columns are known - they are prior months - but the target variable y (revenue) is null. That null is what the model must estimate. The table below illustrates the structure for a fictional mid-size industrial entity.

This is exactly what the gradient boosted models (XGBoost, LightGBM, CatBoost) receive. They do not know what month it is in any human sense - they see numbers in columns. The model learns that when backlog_lag1 is elevated and quote_index is above 1.0, revenue the following month tends to be above average. That pattern, learned from hundreds of rows across the training history, is what produces the forecast.

Model competition, walk-forward validation, and FVA

The Marquis IQ pipeline does not pick a model before seeing the data. It runs all candidate models against each grain independently and lets accuracy determine the winner. The process is walk-forward (expanding window) validation - a rigorous time-series-specific alternative to random cross-validation that respects temporal ordering and prevents data leakage.

Walk-forward validation

In walk-forward validation, the model is trained on the first N months and tested on months N+1 and N+2. Then trained on months 1 through N+1 and tested on N+2 and N+3. The window expands with each step, and accuracy is measured at 1-month and 6-month lead times across all out-of-sample test windows. This mimics the real forecasting environment: the model always trains on the past and predicts the future, never the reverse.

Forecast Value Added (FVA)

The primary model selection criterion is Forecast Value Added: FVA = MAE(Naive) - MAE(Model). A positive FVA means the model outperforms the naive seasonal baseline. A negative FVA means the model is worse than simply repeating last year's same month. FVA is calculated for every model on every grain. The model with the highest positive FVA is selected. If no model beats naive, the naive baseline is used as the forecast - adding complexity for no accuracy gain is not a trade-off worth making.

Why naive is the right floor: the seasonal naive forecast is free, instant, completely transparent, and surprisingly difficult to beat on stable, seasonal businesses. It is not a strawman - it is a genuine benchmark. Any model that earns a negative FVA relative to naive is a liability, not an asset.

Accuracy metrics reported per entity

SHAP: the dollar-level attribution that answers the boardroom question

SHAP (SHapley Additive exPlanations) is a game-theory-based method for attributing a model's prediction to individual input features in a way that is both mathematically rigorous and practically interpretable. For gradient boosted tree models (XGBoost, LightGBM, CatBoost), SHAP values are computed efficiently using TreeSHAP, an algorithm that leverages the tree structure to calculate exact attribution values without approximation.

The question SHAP answers is exactly the boardroom question: of the $487K the model forecasted for next month, how much came from recent revenue history, how much from the backlog position, and how much from quoting activity? Each feature gets a dollar-denominated contribution - positive if it pushed the forecast up, negative if it pulled it down. The baseline (the model's average prediction across all training data) plus the sum of all SHAP values equals the final forecast exactly.

Reading the SHAP output, the answer to "how did it come up with $487K?" is now precise: the model started from a $512K baseline, backlog signals pulled the forecast down by a combined $41K across three lag periods (recent backlog has been running soft), the year trend and 12-month revenue history added back $25K (the business is growing), and a below-average rolling 3-month average removed another $5K. The residual signal from seasonality (Month of Year) added $5K. The sum equals the forecast exactly.

This is the translation layer that makes AI forecasting defensible. A plant controller can read the SHAP output and say "the model is forecasting lower because my backlog position has deteriorated over the last six months - that makes sense." An executive can see that year trend is consistently adding positive contribution across all grains, confirming the business's growth trajectory is being captured. An anomaly - a feature contributing an unexpectedly large positive or negative value - surfaces immediately and triggers investigation.

Forecastability scoring: not everything belongs in an automated forecast

Before the pipeline selects a model for any entity, it runs a forecastability analysis. The question it answers is not "which model is best?" but "should this entity be in an automated forecast at all?" Applying sophisticated statistical models to demand signals that are too sparse, too erratic, or too short to learn from does not produce better forecasts - it produces confident wrong numbers.

Demand classification: the Syntetos-Boylan framework

The pipeline classifies each entity's demand pattern using two measures: ADI (Average Demand Interval), the average number of periods between non-zero demand occurrences, and CV² (squared coefficient of variation), the variance in demand size when demand does occur. Together they define four demand classes:

• Smooth (low ADI, low CV²) - demand arrives regularly with consistent size. Candidate for ETS, XGBoost, LightGBM, CatBoost.
• Erratic (low ADI, high CV²) - demand arrives regularly but in highly variable amounts. ETS with manual review; ML models may overfit.
• Intermittent (high ADI, low CV²) - demand arrives infrequently but consistently sized when it does. Croston's method (SBA variant) is the correct approach.
• Lumpy (high ADI, high CV²) - demand is both infrequent and highly variable. Automated forecasting is not reliable; reorder-point planning is more appropriate.

The classification directly determines which models are candidates for selection. Running XGBoost on a lumpy demand entity wastes compute and produces a forecast no planning team should trust. The forecastability scorecard below shows this classification in action across a sample of entities, with completely anonymized part numbers.

Reading the scorecard top to bottom, the pattern is clear: smooth entities with 40+ months of history and MASE well below 1.0 are strong automated forecast candidates. Erratic entities are forecasted but flagged for manual review before loading into the MRP upload. Intermittent entities route to Croston rather than a standard time-series model. Lumpy entities at the bottom of the list are excluded from the automated forecast entirely and routed to reorder-point planning - a more reliable approach for demand that is both rare and unpredictable.

The Excel output: the accountability layer

The pipeline's final deliverable is not the forecast number. It is an Excel workbook that documents every decision made to produce that number - and gives the planning team the tools to validate it, challenge it, or override it. This workbook is the answer to the boardroom question made tangible.

The workbook contains five tabs that tell the complete story:

• Actuals vs. Forecast - period-by-period actual revenue alongside the model's in-sample fitted values and the out-of-sample forward forecast. The visual gap between actual and fitted is the residual - the part of demand the model did not explain. Large persistent residuals in a specific direction are a signal that something structural changed that the model has not captured.
• FVA by Model and Grain - the full FVA scorecard showing every model's walk-forward accuracy at 1-month and 6-month horizons. The winning model per grain is highlighted. Grains where no model beats naive are flagged in red with a recommendation to investigate data quality or grain definition before the next run.
• SHAP Attribution - the dollar-level feature contribution table for M+1 and M+2 for every grain in the run. This is the tab that answers "how did it come up with that number." Paste rows from this tab into a board deck and the forecast becomes auditable.
• Forecastability Scorecard - every entity classified by demand pattern, scored, and assigned a recommended modeling approach. The planning team uses this tab to decide what goes into the MRP upload versus what gets manual planning intervention.
• Forward Forecast - the actionable output: 12-month rolling forecast by entity, grain, and horizon, ready for validation and MRP upload. This is the tab the procurement manager acts on.

The workbook is regenerated automatically on every pipeline run. Numbers change as new actuals arrive. The SHAP attribution updates to reflect the current feature values. The FVA scorecard re-evaluates model competition with the expanded history. Nothing in the workbook is stale - it is always the current state of the model's understanding of the business.

Related reading: From ERP to MRP: Building a Forecasting Competency That Runs Itself covers the full pipeline architecture. Why Manufacturing Demand Planning Fails covers the data foundation this pipeline requires.