Uniform calibration tests for forecasting systems with small lead time

Bröcker, J. ORCID: https://orcid.org/0000-0002-0864-6530 (2022) Uniform calibration tests for forecasting systems with small lead time. Statistics and Computing, 32. 102. ISSN 0960-3174 doi: 10.1007/s11222-022-10144-9

Abstract/Summary

A long noted difficulty when assessing calibration (or reliability) of forecasting systems is that calibration, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A calibrated probability forecast for binary events for instance should equal the conditional probability of the event given the forecast, whatever the value of the forecast. A new class of tests is presented that are based on estimating the cumulative deviations from calibration. The supremum of those deviations is taken as a test statistic, and the asymptotic distribution of the test statistic is established rigorously. It turns out to be universal, provided the forecasts “look one step ahead” only, or in other words, verify at the next time step in the future. The new tests apply to various different forecasting problems and are compared with established approaches which work in a regression based framework. In comparison to those approaches, the new tests develop power against a wider class of alternatives. Numerical experiments for both artificial data as well as operational weather forecasting systems are presented, and possible extensions to longer lead times are discussed.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/108753
Item Type	Article
Refereed	Yes
Divisions	Science > School of Mathematical, Physical and Computational Sciences > Department of Mathematics and Statistics
Publisher	Springer
Download/View statistics	View download statistics for this item