Evaluating raw ensembles with the continuous ranked probability score

Bröcker, J. (2012) Evaluating raw ensembles with the continuous ranked probability score. Quarterly Journal of the Royal Meteorological Society, 138 (667). pp. 1611-1617. ISSN 1477-870X doi: 10.1002/qj.1891

Abstract/Summary

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.