Reliability, sufficiency, and the decomposition of proper scores

Bröcker, J. (2009) Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135 (643). 1512 - 1519. ISSN 1477-870X doi: 10.1002/qj.456

Abstract/Summary

Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. A scoring rule is called strictly proper if its expectation is optimal if and only if the forecast probability represents the true distribution of the target. In the binary case, strictly proper scoring rules allow for a decomposition into terms related to the resolution and the reliability of a forecast. This fact is particularly well known for the Brier Score. In this article, this result is extended to forecasts for finite-valued targets. Both resolution and reliability are shown to have a positive effect on the score. It is demonstrated that resolution and reliability are directly related to forecast attributes that are desirable on grounds independent of the notion of scores. This finding can be considered an epistemological justification of measuring forecast quality by proper scoring rules. A link is provided to the original work of DeGroot and Fienberg, extending their concepts of sufficiency and refinement. The relation to the conjectured sharpness principle of Gneiting, et al., is elucidated.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/29154
Item Type	Article
Refereed	Yes
Divisions	No Reading authors. Back catalogue items Science > School of Mathematical, Physical and Computational Sciences > Department of Mathematics and Statistics Science > School of Mathematical, Physical and Computational Sciences > Department of Meteorology
Uncontrolled Keywords	probabilistic forecasts, scoring rules, reliability, resolution
Publisher	Royal Meteorological Society
Download/View statistics	View download statistics for this item