Stimulus-specific random effects inflate false-positive classification accuracy in multivariate-voxel-pattern-analysis: a solution with generalized mixed-effects modelling

Kajimura, S., Hoshino, T. and Murayama, K. (2023) Stimulus-specific random effects inflate false-positive classification accuracy in multivariate-voxel-pattern-analysis: a solution with generalized mixed-effects modelling. NeuroImage, 269. 119901. ISSN 1053-8119 doi: 10.1016/j.neuroimage.2023.119901

Abstract/Summary

When conducting multivariate-voxel pattern analysis (MVPA), researchers typically compute the average accuracy for each subject and statistically test if the average accuracy is different from the chance level across subjects (by-subject analysis). We argue that this traditional by-subject analysis leads to inflated Type-1 error rates, regardless of the type of machine learning method used (e.g., support vector machine). This is because by-subject analysis does not consider the variance attributed to the idiosyncratic features of the stimuli that have a common influence on all subjects (i.e., the random stimulus effect). As a solution, we proposed the use of generalized linear mixed-effects modelling to evaluate average accuracy. This method only requires post-classification data (i.e., it does not consider the type of classification methods used) and is easily implemented in the analysis pipeline with common statistical software (SPSS, R, Python, etc.). Using both statistical simulation and real fMRI data analysis, we demonstrated that the traditional by-subject method indeed increases Type-1 error rates to a considerable degree, while generalized mixed-effects modelling that incorporates random stimulus effects can indeed maintain the nominal Type-1 error rates.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/110298
Identification Number/DOI	10.1016/j.neuroimage.2023.119901
Refereed	Yes
Divisions	Life Sciences > School of Psychology and Clinical Language Sciences > Department of Psychology
Publisher	Elsevier
Download/View statistics	View download statistics for this item