High‐dimensional covariance estimation from a small number of samples

Vishny, D., Morzfeld, M., Gwirtz, K., Bach, E. ORCID: https://orcid.org/0000-0002-9725-0203, Dunbar, O. R. A. and Hodyss, D. (2024) High‐dimensional covariance estimation from a small number of samples. Journal of Advances in Modeling Earth Systems, 16 (9). e2024MS004417. ISSN 1942-2466 doi: 10.1029/2024MS004417

Abstract/Summary

We synthesize knowledge from numerical weather prediction, inverse theory, and statistics to address the problem of estimating a high-dimensional covariance matrix from a small number of samples. This problem is fundamental in statistics, machine learning/artificial intelligence, and in modern Earth science. We create several new adaptive methods for high-dimensional covariance estimation, but one method, which we call Noise-Informed Covariance Estimation (NICE), stands out because it has three important properties: (a) NICE is conceptually simple and computationally efficient; (b) NICE guarantees symmetric positive semi-definite covariance estimates; and (c) NICE is largely tuning-free. We illustrate the use of NICE on a large set of Earth science–inspired numerical examples, including cycling data assimilation, inversion of geophysical field data, and training of feed-forward neural networks with time-averaged data from a chaotic dynamical system. Our theory, heuristics and numerical tests suggest that NICE may indeed be a viable option for high-dimensional covariance estimation in many Earth science problems.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/117933
Identification Number/DOI	10.1029/2024MS004417
Refereed	Yes
Divisions	Science > School of Mathematical, Physical and Computational Sciences > Department of Mathematics and Statistics Science > School of Mathematical, Physical and Computational Sciences > Department of Meteorology
Publisher	American Geophysical Union
Download/View statistics	View download statistics for this item