Benchmark
What the engine scored, measured rather than asserted — and the three things that make that number worse than it first looks.
Last updated
The result
On one real catalogue of about 4,000 SKUs, measured 25 August 2026:
levelin beat a four-week moving average on 42% of products, tied
on
18%, and lost on 40%. Median MASE
0.877 against the baseline's 0.938.
That is roughly 3 to 5 percent lower average error, and materially fewer catastrophic misses. It is not "more accurate on your SKUs", and we are not going to say that it is.
Three things that make it worse
These are set at the same size as the number above, because that is the whole point.
One — the denominator is not your catalogue
Measured across the whole catalogue rather than the scored part, the figure is 37.0%, not 42%. The 42% excludes SKUs with no demand in the test window.
Two — a third of the catalogue was never scored at all
33.5% of SKUs had too little sales history to backtest — including every SKU the classifier could not place, a quarter of that catalogue on its own. Those fall through to a routing default with nothing measured behind them. In the app they are marked Unproven, which is what that word means.
Three — the winning method graded its own homework
In the headline run, the method chosen per SKU was selected on the same
windows it was then scored on. Re-run with the champion selected on earlier
windows and scored on one it had never seen, the result holds — that is the
0.877 quoted above — but the original 55.7% figure this
project published in an earlier post did not survive the correction. It is written up in full, including what we got wrong.
Where it is weakest
Split the catalogue by what demand is doing rather than by its pattern class, and the engine is near-perfect on stock that is dying and close to useless on stock that is taking off.
| Cohort | Median MASE | Beating naive |
|---|---|---|
| Steady | 0.80 | 60% |
| Shrinking | 0.27 | 80% |
| Growing | 1.46 | 35% |
| New (first sale ≤16 weeks) | 2.57 | 5% |
A stockout on a ramping product is the expensive failure, and it is exactly the one this family of methods cannot see: every model levelin routes to emits a flat path, with no trend term and no seasonality term anywhere in the engine. No model we tested, in the routing table or out of it, fixes it. Detecting it and saying so is the only honest response available, and it is what we are building next.
Method
- Dataset: UCI Online Retail II — a real UK online retailer, roughly one million transactions, 4,984 SKUs, December 2009 to December 2011. Cancellations and negative quantities excluded.
- Buckets: weekly, 52 periods, four-period horizon, three rolling cross-validation windows.
- Metric: MASE with seasonality 1 — error divided by what a "same as last period" forecast scores on the same series. Below 1.00 beats that baseline. wMAPE reported alongside.
- Baselines: last-value, four-week moving average, historic average, scored on identical folds over the identical SKU set.
- Reproducible: the comparison script lives in the repository and is run by hand before any accuracy claim is published. It is deliberately not a CI gate, because the dataset is a 45 MB file that is not in the repository.
The limit that matters most
This is one dataset. A UK online giftware retailer with an unusually strong Christmas ramp. It tests whether the result survives changes of configuration, not whether it survives a different catalogue. Yours would be the second catalogue this engine has ever been measured on — and we will tell you what it scored.
Sources
Every competitor claim on this site, with where it came from and when it was checked. Vendors change their pricing and feature pages; if one of these has gone stale, tell us and we will correct it here.
| Claim | Source | Checked |
|---|---|---|
| Stocky stops working 31 August 2026 | Shopify Help Centre | 25 Aug 2026 |
| Shopify's native inventory features exclude forecasting | Shopify | 25 Aug 2026 |
| Assisty documents comparing forecast to actual by hand | App Store listing and vendor docs | 25 Aug 2026 |
| Prediko's entry plan starts at $49 and bands by revenue | App Store listing | 25 Aug 2026 |
| Inventory Planner is quote-only above its entry tier | App Store listing | 25 Aug 2026 |
| No app examined exposes per-SKU forecast error | Read directly: Inventory Planner, Prediko, Assisty, Cogsy, Stocky | 25 Aug 2026 |
The last row is the strongest claim on this site and the hardest to prove: it is an absence, established by reading five products' own documentation. If you work on one of these and it is wrong, we would genuinely like to know.