Psychology Department Colloquium

Uri Simonsohn

Operations and Information Management Department

The Wharton School, University of Pennsylvania

 Reception to Folllow - Tillett Room 102

"p-curve: A key to the file-drawer"

abstract:
We propose using "p-curve," the distribution of significant p-values behind any set of findings, to assess whether they are likely to replicate, and whether they have been "p-hacked" (analyses were chosen just to get statistical significance). We show that p-curve can be meaningfully analyzed when arising from small sets of p-values, say those present in a single paper. We illustrate its use comparing p-curves for a set of psychology experiments that we expected to have been p-hacked, and for a set we expected not to have been. P-curve can be used to aggregate the evidential value of disparate sets of findings, such as those by a given author, journal or institution. Finally, the p-curve of a set of published findings can be analyzed, ignoring all non-published ones, to obtain a 100% publication-bias-free effect-size estimate.

More Info:

Uri Simonsohn, Wharton, is at the forefront of the new Scientific Integrity Movement in Psychology.  He has demonstrated that several practices common in psychological
research lead to invalid conclusions; and has developed techniques capable of revealing data irregularities and even fraud.  Those techniques have already led to two psychologists
to resign under clouds of suspicion for fraud.  This talk will describe a new method for identifying "p-hacking," which refers to analyses and procedures selected in order to reach
the almighty p<.05.  Such analyses and procedures, though often useful for convincing editors and reviewers that a study is worthwhile, actually  invalidate the conclusions based on studies
using such techniques.

http://www.nature.com/news/the-data-detective-1.10937

http://news.sciencemag.org/scienceinsider/2012/06/fraud-detection-method-called-cr.html