Meta-analysis to integrate effect sizes within a paper: possible misuse and Type-1 error inflation

Ueno, T., Fastrich, G. and Murayama, K. (2016) Meta-analysis to integrate effect sizes within a paper: possible misuse and Type-1 error inflation. Journal of Experimental Psychology: General, 145 (5). pp. 643-654. ISSN 1939-2222 doi: 10.1037/xge0000159

Abstract/Summary

In recent years an increasing number of papers have employed meta-analysis to integrate effect sizes of researchers’ own series of studies within a single paper (“internal meta-analysis”). Although this approach has the obvious advantage of obtaining narrower confidence intervals, we show that it could inadvertently inflate false-positive rates if researchers are motivated to use internal meta-analysis in order to obtain a significant overall effect. Specifically, if one decides whether to stop or continue a further replication experiment depending on the significance of the results in an internal meta-analysis, false-positive rates would increase beyond the nominal level. We conducted a set of Monte-Carlo simulations to demonstrate our argument, and provided a literature review to gauge awareness and prevalence of this issue. Furthermore, we made several recommendations when using internal meta-analysis to make a judgment on statistical significance.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/58430
Identification Number/DOI	10.1037/xge0000159
Refereed	Yes
Divisions	Life Sciences > School of Psychology and Clinical Language Sciences > Department of Psychology
Publisher	American Psychological Association
Download/View statistics	View download statistics for this item