Beyond Point Accuracy: Evaluating Probabilistic Machine-Learning Cash-Flow Forecasts for Small and Medium-Sized Enterprises

Abstract

Machine-learning cash-flow forecasting is increasingly offered to small and medium-sized enterprises (SMEs) as part of AI-driven financial planning and analysis [1]. Most evaluations of such tools report only point accuracy, yet SME owners use forecasts to answer a different question: how likely is the business to run short of cash, and how early can it know? This article proposes an evaluation framework that judges SME cash-flow forecasts on three dimensions together: scaled accuracy, calibration of prediction intervals, and decision value measured through cash-shortfall alerts. We demonstrate the framework in a reproducible experiment on 300 simulated SMEs with weekly cash flows over three years, comparing a naive baseline, a seasonal-naive baseline, a gradient-boosted quantile model and the same model with conformal calibration. The machine-learning model matched but did not beat the seasonal-naive baseline on point accuracy (relative mean absolute error 0.39 for both, against 1.00 for the naive method). Its raw 80 percent prediction intervals covered only 61 percent of outcomes; conformal calibration raised coverage to 73 percent, with the remaining gap traced to demand shocks absent from the calibration window. For shortfall alerts, the naive method detected the most shortfalls but raised false alarms in 25 percent of safe cases, against 6 to 7 percent for the better methods. We then replicate the evaluation on real weekly revenue of 327 small online sellers from a public Brazilian marketplace dataset. There the ranking of methods changed: the seasonal-naive benchmark failed (relative MAE 1.94) because the sellers were still growing, the machine-learning model was most accurate (0.97) and had the lowest pinball loss, its raw 80 percent intervals again covered only 65 percent of outcomes, and conformal calibration restored coverage to 82 percent. A shortfall alert based on the lower 10 percent quantile, which worked in simulation, raised false alarms in most safe cases on these intermittent revenue series. The results show that accuracy alone can mislead SME adoption decisions and that calibration and decision-oriented metrics should be standard in evaluating AI-driven financial planning tools.