Type-1 error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective

Murayama, K., Sakaki, M. ORCID: https://orcid.org/0000-0003-1993-5765, Yan, V. X. and Smith, G. (2014) Type-1 error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective. Journal of Experimental Psychology: Learning, Memory & Cognition, 40 (5). pp. 1287-1306. ISSN 0278-7393 doi: 10.1037/a0036914

Abstract/Summary

In order to examine metacognitive accuracy (i.e., the relationship between metacognitive judgment and memory performance), researchers often rely on by-participant analysis, where metacognitive accuracy (e.g., resolution, as measured by the gamma coefficient or signal detection measures) is computed for each participant and the computed values are entered into group-level statistical tests such as the t-test. In the current work, we argue that the by-participant analysis, regardless of the accuracy measurements used, would produce a substantial inflation of Type-1 error rates, when a random item effect is present. A mixed-effects model is proposed as a way to effectively address the issue, and our simulation studies examining Type-1 error rates indeed showed superior performance of mixed-effects model analysis as compared to the conventional by-participant analysis. We also present real data applications to illustrate further strengths of mixed-effects model analysis. Our findings imply that caution is needed when using the by-participant analysis, and recommend the mixed-effects model analysis.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/36409
Identification Number/DOI	10.1037/a0036914
Refereed	Yes
Divisions	Science Life Sciences > School of Psychology and Clinical Language Sciences > Department of Psychology Life Sciences > School of Psychology and Clinical Language Sciences > Language and Cognition
Publisher	American Psychological Association.
Publisher Statement	This article may not exactly replicate the final version published in the APA journal. It is not the copy of record
Download/View statistics	View download statistics for this item