User menu

Accès à distance ? S'identifier sur le proxy UCLouvain

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Primary tabs

download
  • Open access
  • PDF
  • 961.18 K
download
  • Open access
  • PDF
  • 119.72 K
Klein, Richard [Université Grenoble Alpes, France] Cambier, Fanny [UCL] Kervyn de Meerendré, Nicolas [UCL]

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Document type Article de périodique (Journal article) – Article de recherche
Access type Accès libre
Publication date 2018
Language Anglais
Journal information "Advances in methods and practices in psychological science" - Vol. 1, no.4, p. 443-490 (2018)
Peer reviewed yes
Publisher Sage Publications
issn 2515-2459
e-issn 2515-2467
Publication status Publié
Affiliation UCL - SSH/LouRIM - Louvain Research Institute in Management and Organizations
Keywords social psychology ; cognitive psychology ; replication ; culture ; individual differences ; sampling effects ; situational effects ; meta-analysis ; Registered Report ; open data ; open materials ; preregistered
Links
  1. Adler Nancy E., Boyce Thomas, Chesney Margaret A., Cohen Sheldon, Folkman Susan, Kahn Robert L., Syme S. Leonard, Socioeconomic status and health: The challenge of the gradient., 10.1037/0003-066x.49.1.15
  2. Adler Nancy E., Epel Elissa S., Castellazzo Grace, Ickovics Jeannette R., Relationship of subjective and objective social status with psychological and physiological functioning: Preliminary data in healthy, White women., 10.1037/0278-6133.19.6.586
  3. Alter Adam L., Oppenheimer Daniel M., Epley Nicholas, Eyre Rebecca N., Overcoming intuition: Metacognitive difficulty activates analytic reasoning., 10.1037/0096-3445.136.4.569
  4. Anderson Cameron, Kraus Michael W., Galinsky Adam D., Keltner Dacher, The Local-Ladder Effect : Social Status and Subjective Well-Being, 10.1177/0956797611434537
  5. Ashton-James Claire E., Maddux William W., Galinsky Adam D., Chartrand Tanya L., Who I Am Depends on How I Feel : The Role of Affect in the Expression of Culture, 10.1111/j.1467-9280.2009.02299.x
  6. Bakker Marjan, Hartgerink Chris H. J., Wicherts Jelte M., van der Maas Han L. J., Researchers’ Intuitions About Power in Psychological Research, 10.1177/0956797616647519
  7. Bauer Monika A., Wilkie James E. B., Kim Jung K., Bodenhausen Galen V., Cuing Consumerism : Situational Materialism Undermines Personal and Social Well-Being, 10.1177/0956797611429579
  8. Baumeister Roy F., Vohs Kathleen D., Misguided Effort With Elusive Implications, 10.1177/1745691616652878
  9. Benjamin Daniel J., Berger James O., Johannesson Magnus, Nosek Brian A., Wagenmakers E.-J., Berk Richard, Bollen Kenneth A., Brembs Björn, Brown Lawrence, Camerer Colin, Cesarini David, Chambers Christopher D., Clyde Merlise, Cook Thomas D., De Boeck Paul, Dienes Zoltan, Dreber Anna, Easwaran Kenny, Efferson Charles, Fehr Ernst, Fidler Fiona, Field Andy P., Forster Malcolm, George Edward I., Gonzalez Richard, Goodman Steven, Green Edwin, Green Donald P., Greenwald Anthony G., Hadfield Jarrod D., Hedges Larry V., Held Leonhard, Hua Ho Teck, Hoijtink Herbert, Hruschka Daniel J., Imai Kosuke, Imbens Guido, Ioannidis John P. A., Jeon Minjeong, Jones James Holland, Kirchler Michael, Laibson David, List John, Little Roderick, Lupia Arthur, Machery Edouard, Maxwell Scott E., McCarthy Michael, Moore Don A., Morgan Stephen L., Munafó Marcus, Nakagawa Shinichi, Nyhan Brendan, Parker Timothy H., Pericchi Luis, Perugini Marco, Rouder Jeff, Rousseau Judith, Savalei Victoria, Schönbrodt Felix D., Sellke Thomas, Sinclair Betsy, Tingley Dustin, Van Zandt Trisha, Vazire Simine, Watts Duncan J., Winship Christopher, Wolpert Robert L., Xie Yu, Young Cristobal, Zinman Jonathan, Johnson Valen E., Redefine statistical significance, 10.1038/s41562-017-0189-z
  10. Birnbaum Michael H., Human Research and Data Collection via the Internet, 10.1146/annurev.psych.55.090902.141601
  11. Borenstein Michael, Hedges Larry V., Higgins Julian P. T., Rothstein Hannah R., Introduction to Meta-Analysis, ISBN:9780470743386, 10.1002/9780470743386
  12. Brislin Richard W., Back-Translation for Cross-Cultural Research, 10.1177/135910457000100301
  13. Button Katherine S., Ioannidis John P. A., Mokrysz Claire, Nosek Brian A., Flint Jonathan, Robinson Emma S. J., Munafò Marcus R., Power failure: why small sample size undermines the reliability of neuroscience, 10.1038/nrn3475
  14. Camerer C. F., Dreber A., Forsell E., Ho T.-H., Huber J., Johannesson M., Kirchler M., Almenberg J., Altmejd A., Chan T., Heikensten E., Holzmeister F., Imai T., Isaksson S., Nave G., Pfeiffer T., Razen M., Wu H., Evaluating replicability of laboratory experiments in economics, 10.1126/science.aaf0918
  15. Camerer Colin F., Dreber Anna, Holzmeister Felix, Ho Teck-Hua, Huber Jürgen, Johannesson Magnus, Kirchler Michael, Nave Gideon, Nosek Brian A., Pfeiffer Thomas, Altmejd Adam, Buttrick Nick, Chan Taizan, Chen Yiling, Forsell Eskil, Gampa Anup, Heikensten Emma, Hummer Lily, Imai Taisuke, Isaksson Siri, Manfredi Dylan, Rose Julia, Wagenmakers Eric-Jan, Wu Hang, Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015, 10.1038/s41562-018-0399-z
  16. Chandler Jesse, Mueller Pam, Paolacci Gabriele, Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers, 10.3758/s13428-013-0365-7
  17. Cheung Felix, Lucas Richard E., Assessing the validity of single-item life satisfaction measures: results from three large samples, 10.1007/s11136-014-0726-4
  18. Cohen Geoffrey L., Sherman David K., Bastardi Anthony, Hsu Lillian, McGoey Michelle, Ross Lee, Bridging the partisan divide: Self-affirmation reduces ideological closed-mindedness and inflexibility in negotiation., 10.1037/0022-3514.93.3.415
  19. Cohen Sheldon, Alper Cuneyt M., Doyle William J., Adler Nancy, Treanor John J., Turner Ronald B., Objective and subjective socioeconomic status and susceptibility to the common cold., 10.1037/0278-6133.27.2.268
  20. Coppock A., Political Science Research Methods
  21. Cramer Angélique O. J., van Ravenzwaaij Don, Matzke Dora, Steingroever Helen, Wetzels Ruud, Grasman Raoul P. P. P., Waldorp Lourens J., Wagenmakers Eric-Jan, Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies, 10.3758/s13423-015-0913-5
  22. Monin Benoît, Oppenheimer Daniel M., Ferguson Melissa J., Carter Travis J., Hassin Ran R., Crisp Richard J., Miles Eleanor, Husnu Shenel, Schwarz Norbert, Strack Fritz, Klein Richard A., Ratliff Kate A., Vianello Michelangelo, Adams Reginald B., Bahník Štěpán, Bernstein Michael J., Bocian Konrad, Brandt Mark J., Brooks Beach, Chloe Brumbaugh Claudia, Cemalcilar Zeynep, Chandler Jesse, Cheong Winnee, Davis William E., Devos Thierry, Eisner Matthew, Frankowska Natalia, Furrow David, Maria Galliani Elisa, Hasselman Fred, Hicks Joshua A., Hovermale James F., Jane Hunt S., Huntsinger Jeffrey R., IJzerman Hans, John Melissa-Sue, Joy-Gaba Jennifer A., Barry Kappes Heather, Krueger Lacy E., Kurtz Jaime, Levitan Carmel A., Mallett Robyn K., Morris Wendy L., Nelson Anthony J., Nier Jason A., Packard Grant, Pilati Ronaldo, Rutchick Abraham M., Schmidt Kathleen, Skorinko Jeanine L., Smith Robert, Steiner Troy G., Storbeck Justin, Van Swol Lyn M., Thompson Donna, van ‘t Veer A. E., Ann Vaughn Leigh, Vranka Marek, Wichman Aaron L., Woodzicka Julie A., Nosek Brian A., Kahneman Daniel, Commentaries and Rejoinder on, 10.1027/1864-9335/a000202
  23. Critcher Clayton R., Gilovich Thomas, Incidental environmental anchors, 10.1002/bdm.586
  24. Dandurand Frédéric, Shultz Thomas R., Onishi Kristine H., Comparing online and lab methods in a problem-solving experiment, 10.3758/brm.40.2.428
  25. Doyen Stéphane, Klein Olivier, Pichon Cora-Lise, Cleeremans Axel, Behavioral Priming: It's All in the Mind, but Whose Mind?, 10.1371/journal.pone.0029081
  26. Dreber Anna, Pfeiffer Thomas, Almenberg Johan, Isaksson Siri, Wilson Brad, Chen Yiling, Nosek Brian A., Johannesson Magnus, Using prediction markets to estimate the reproducibility of scientific research, 10.1073/pnas.1516179112
  27. Ebersole Charles R., Atherton Olivia E., Belanger Aimee L., Skulborstad Hayley M., Allen Jill M., Banks Jonathan B., Baranski Erica, Bernstein Michael J., Bonfiglio Diane B.V., Boucher Leanne, Brown Elizabeth R., Budiman Nancy I., Cairo Athena H., Capaldi Colin A., Chartier Christopher R., Chung Joanne M., Cicero David C., Coleman Jennifer A., Conway John G., Davis William E., Devos Thierry, Fletcher Melody M., German Komi, Grahe Jon E., Hermann Anthony D., Hicks Joshua A., Honeycutt Nathan, Humphrey Brandon, Janus Matthew, Johnson David J., Joy-Gaba Jennifer A., Juzeler Hannah, Keres Ashley, Kinney Diana, Kirshenbaum Jacqeline, Klein Richard A., Lucas Richard E., Lustgraaf Christopher J.N., Martin Daniel, Menon Madhavi, Metzger Mitchell, Moloney Jaclyn M., Morse Patrick J., Prislin Radmila, Razza Timothy, Re Daniel E., Rule Nicholas O., Sacco Donald F., Sauerberger Kyle, Shrider Emily, Shultz Megan, Siemsen Courtney, Sobocko Karin, Weylin Sternglanz R., Summerville Amy, Tskhay Konstantin O., van Allen Zack, Vaughn Leigh Ann, Walker Ryan J., Weinberg Ashley, Wilson John Paul, Wirth James H., Wortman Jessica, Nosek Brian A., Many Labs 3: Evaluating participant pool quality across the academic semester via replication, 10.1016/j.jesp.2015.10.012
  28. Ehrhart Mark G., Ehrhart Karen Holcombe, Roesch Scott C., Chung-Herrera Beth G., Nadler Kristy, Bradshaw Kelsey, Testing the latent factor structure and construct validity of the Ten-Item Personality Inventory, 10.1016/j.paid.2009.07.012
  29. Monin Benoît, Oppenheimer Daniel M., Ferguson Melissa J., Carter Travis J., Hassin Ran R., Crisp Richard J., Miles Eleanor, Husnu Shenel, Schwarz Norbert, Strack Fritz, Klein Richard A., Ratliff Kate A., Vianello Michelangelo, Adams Reginald B., Bahník Štěpán, Bernstein Michael J., Bocian Konrad, Brandt Mark J., Brooks Beach, Chloe Brumbaugh Claudia, Cemalcilar Zeynep, Chandler Jesse, Cheong Winnee, Davis William E., Devos Thierry, Eisner Matthew, Frankowska Natalia, Furrow David, Maria Galliani Elisa, Hasselman Fred, Hicks Joshua A., Hovermale James F., Jane Hunt S., Huntsinger Jeffrey R., IJzerman Hans, John Melissa-Sue, Joy-Gaba Jennifer A., Barry Kappes Heather, Krueger Lacy E., Kurtz Jaime, Levitan Carmel A., Mallett Robyn K., Morris Wendy L., Nelson Anthony J., Nier Jason A., Packard Grant, Pilati Ronaldo, Rutchick Abraham M., Schmidt Kathleen, Skorinko Jeanine L., Smith Robert, Steiner Troy G., Storbeck Justin, Van Swol Lyn M., Thompson Donna, van ‘t Veer A. E., Ann Vaughn Leigh, Vranka Marek, Wichman Aaron L., Woodzicka Julie A., Nosek Brian A., Kahneman Daniel, Commentaries and Rejoinder on, 10.1027/1864-9335/a000202
  30. Finucane Melissa L., Gullion Christina M., Developing a tool for measuring the decision-making competence of older adults., 10.1037/a0019106
  31. Forsell Eskil, Viganola Domenico, Pfeiffer Thomas, Almenberg Johan, Wilson Brad, Chen Yiling, Nosek Brian A., Johannesson Magnus, Dreber Anna, Predicting replication outcomes in the Many Labs 2 study, 10.1016/j.joep.2018.10.009
  32. Fraley R. Chris, Vazire Simine, The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power, 10.1371/journal.pone.0109019
  33. Frazier Patricia A., Tix Andrew P., Barron Kenneth E., Testing Moderator and Mediator Effects in Counseling Psychology Research., 10.1037/0022-0167.51.1.115
  34. Frederick Shane, Cognitive Reflection and Decision Making, 10.1257/089533005775196732
  35. Giessner Steffen R., Schubert Thomas W., High in the hierarchy: How vertical location and judgments of leaders’ power are interrelated, 10.1016/j.obhdp.2006.10.001
  36. Gilbert D. T., King G., Pettigrew S., Wilson T. D., Comment on "Estimating the reproducibility of psychological science", 10.1126/science.aad7243
  37. Gilbert Daniel T., Malone Patrick S., The correspondence bias., 10.1037/0033-2909.117.1.21
  38. Gnambs Timo, A meta-analysis of dependability coefficients (test–retest reliabilities) for measures of the Big Five, 10.1016/j.jrp.2014.06.003
  39. Goldberg L. R., Review of personality and social psychology, 2, 141 (1981)
  40. Gosling Samuel D, Rentfrow Peter J, Swann William B, A very brief measure of the Big-Five personality domains, 10.1016/s0092-6566(03)00046-1
  41. Graham Jesse, Haidt Jonathan, Nosek Brian A., Liberals and conservatives rely on different sets of moral foundations., 10.1037/a0015141
  42. Gray Kurt, Wegner Daniel M., Moral typecasting: Divergent perceptions of moral agents and moral patients., 10.1037/a0013748
  43. Greenwald Anthony G., Consequences of prejudice against the null hypothesis., 10.1037/h0076157
  44. Hagger M. S., Chatzisarantis N. L. D., Alberts H., Anggono C. O., Batailler C., Birt A. R., Brand R., Brandt M. J., Brewer G., Bruyneel S., Calvillo D. P., Campbell W. K., Cannon P. R., Carlucci M., Carruth N. P., Cheung T., Crowell A., De Ridder D. T. D., Dewitte S., Elson M., Evans J. R., Fay B. A., Fennis B. M., Finley A., Francis Z., Heise E., Hoemann H., Inzlicht M., Koole S. L., Koppel L., Kroese F., Lange F., Lau K., Lynch B. P., Martijn C., Merckelbach H., Mills N. V., Michirev A., Miyake A., Mosser A. E., Muise M., Muller D., Muzi M., Nalis D., Nurwanti R., Otgaar H., Philipp M. C., Primoceri P., Rentzsch K., Ringos L., Schlinkert C., Schmeichel B. J., Schoch S. F., Schrama M., Schütz A., Stamos A., Tinghög G., Ullrich J., vanDellen M., Wimbarti S., Wolff W., Yusainy C., Zerhouni O., Zwienenberg M., A Multilab Preregistered Replication of the Ego-Depletion Effect, 10.1177/1745691616652873
  45. Haidt Jonathan, McCauley Clark, Rozin Paul, Individual differences in sensitivity to disgust: A scale sampling seven domains of disgust elicitors, 10.1016/0191-8869(94)90212-7
  46. HAUSER MARC, CUSHMAN FIERY, YOUNG LIANE, KANG-XING JIN R., MIKHAIL JOHN, A Dissociation Between Moral Judgments and Justifications, 10.1111/j.1468-0017.2006.00297.x
  47. Hawkins Carlee Beth, Fitzgerald Cailey E., Nosek Brian A., In Search of an Association Between Conception Risk and Prejudice, 10.1177/0956797614553121
  48. Henrich Joseph, Heine Steven J., Norenzayan Ara, The weirdest people in the world?, 10.1017/s0140525x0999152x
  49. Hilbig Benjamin E., Reaction time effects in lab- versus Web-based research: Experimental evidence, 10.3758/s13428-015-0678-9
  50. Hsee Christopher K., Less is better: when low-value options are valued more highly than high-value options, 10.1002/(sici)1099-0771(199806)11:2<107::aid-bdm292>3.0.co;2-y
  51. Huang Yanli, Tse Chi-Shing, Cho Kit W., Living in the north is not necessarily favorable: Different metaphoric associations between cardinal direction and valence in Hong Kong and in the United States : Metaphoric Association, 10.1002/ejsp.2013
  52. Inbar Yoel, Pizarro David A., Knobe Joshua, Bloom Paul, Disgust sensitivity predicts intuitive disapproval of gays., 10.1037/a0015960
  53. John Leslie K., Loewenstein George, Prelec Drazen, Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling, 10.1177/0956797611430953
  54. Johnson David J., Cheung Felix, Donnellan M. Brent, Does Cleanliness Influence Moral Judgments? : A Direct Replication of, 10.1027/1864-9335/a000186
  55. Jones Edward E, Harris Victor A, The attribution of attitudes, 10.1016/0022-1031(67)90034-0
  56. Kahneman Daniel, Commentary on Ebersole et al. (2016), 10.1016/j.jesp.2016.04.002
  57. Kay Aaron C., Laurin Kristin, Fitzsimons Gráinne M., Landau Mark J., A functional basis for structure-seeking: Exposure to structure promotes willingness to engage in motivated action., 10.1037/a0034462
  58. Klein Richard A., Ratliff Kate A., Vianello Michelangelo, Adams Reginald B., Bahník Štěpán, Bernstein Michael J., Bocian Konrad, Brandt Mark J., Brooks Beach, Brumbaugh Claudia Chloe, Cemalcilar Zeynep, Chandler Jesse, Cheong Winnee, Davis William E., Devos Thierry, Eisner Matthew, Frankowska Natalia, Furrow David, Galliani Elisa Maria, Hasselman Fred, Hicks Joshua A., Hovermale James F., Hunt S. Jane, Huntsinger Jeffrey R., IJzerman Hans, John Melissa-Sue, Joy-Gaba Jennifer A., Barry Kappes Heather, Krueger Lacy E., Kurtz Jaime, Levitan Carmel A., Mallett Robyn K., Morris Wendy L., Nelson Anthony J., Nier Jason A., Packard Grant, Pilati Ronaldo, Rutchick Abraham M., Schmidt Kathleen, Skorinko Jeanine L., Smith Robert, Steiner Troy G., Storbeck Justin, Van Swol Lyn M., Thompson Donna, van ‘t Veer A. E., Ann Vaughn Leigh, Vranka Marek, Wichman Aaron L., Woodzicka Julie A., Nosek Brian A., Investigating Variation in Replicability : A “Many Labs” Replication Project, 10.1027/1864-9335/a000178
  59. Klein Richard A., Ratliff Kate A., Vianello Michelangelo, Adams Reginald B., Bahník Štěpán, Bernstein Michael J., Bocian Konrad, Brandt Mark J., Brooks Beach, Brumbaugh Claudia Chloe, Cemalcilar Zeynep, Chandler Jesse, Cheong Winnee, Davis William E., Devos Thierry, Eisner Matthew, Frankowska Natalia, Furrow David, Galliani Elisa Maria, Hasselman Fred, Hicks Joshua A., Hovermale James F., Hunt S. Jane, Huntsinger Jeffrey R., IJzerman Hans, John Melissa-Sue, Joy-Gaba Jennifer A., Barry Kappes Heather, Krueger Lacy E., Kurtz Jaime, Levitan Carmel A., Mallett Robyn K., Morris Wendy L., Nelson Anthony J., Nier Jason A., Packard Grant, Pilati Ronaldo, Rutchick Abraham M., Schmidt Kathleen, Skorinko Jeanine L., Smith Robert, Steiner Troy G., Storbeck Justin, Van Swol Lyn M., Thompson Donna, van ‘t Veer A. E., Ann Vaughn Leigh, Vranka Marek, Wichman Aaron L., Woodzicka Julie A., Nosek Brian A., Investigating Variation in Replicability : A “Many Labs” Replication Project, 10.1027/1864-9335/a000178
  60. Knobe Joshua, Intentional action and side effects in ordinary language, 10.1111/1467-8284.00419
  61. Knobe Joshua, The Concept of Intentional Action: A Case Study in the Uses of Folk Psychology, 10.1007/s11098-004-4510-0
  62. Krupnikov Yanna, Levine Adam Seth, Cross-Sample Comparisons and External Validity, 10.1017/xps.2014.7
  63. Lewin Kurt, Principles of topological psychology., 10.1037/10019-000
  64. Lucas Richard E., Brent Donnellan M., Estimating the Reliability of Single-Item Life Satisfaction Measures: Results from Four National Panel Studies, 10.1007/s11205-011-9783-z
  65. Markus Hazel R., Kitayama Shinobu, Culture and the self: Implications for cognition, emotion, and motivation., 10.1037/0033-295x.98.2.224
  66. McClelland Gary H., Optimal design in psychological research., 10.1037/1082-989x.2.1.3
  67. Meade Adam W., Craig S. Bartholomew, Identifying careless responses in survey data., 10.1037/a0028085
  68. Miguel E., Camerer C., Casey K., Cohen J., Esterling K. M., Gerber A., Glennerster R., Green D. P., Humphreys M., Imbens G., Laitin D., Madon T., Nelson L., Nosek B. A., Petersen M., Sedlmayr R., Simmons J. P., Simonsohn U., Van der Laan M., Promoting Transparency in Social Science Research, 10.1126/science.1245317
  69. Miyamoto Yuri, Kitayama Shinobu, Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior., 10.1037/0022-3514.83.5.1239
  70. Mullen Brian, Atkins Jennifer L, Champion Debbie S, Edwards Cecelia, Hardy Dana, Story John E, Vanderklok Mary, The false consensus effect: A meta-analysis of 115 hypothesis tests, 10.1016/0022-1031(85)90020-4
  71. Mullinix Kevin J., Leeper Thomas J., Druckman James N., Freese Jeremy, The Generalizability of Survey Experiments, 10.1017/xps.2015.19
  72. Murphy Ryan O., Ackermann Kurt A., Handgraaf Michel, Measuring Social Value Orientation, 10.2139/ssrn.1804189
  73. Norenzayan Ara, Smith Edward E., Kim Beom Jun, Nisbett Richard E., Cultural preferences for formal versus intuitive reasoning, 10.1207/s15516709cog2605_4
  74. Nosek B. A., Alter G., Banks G. C., Borsboom D., Bowman S. D., Breckler S. J., Buck S., Chambers C. D., Chin G., Christensen G., Contestabile M., Dafoe A., Eich E., Freese J., Glennerster R., Goroff D., Green D. P., Hesse B., Humphreys M., Ishiyama J., Karlan D., Kraut A., Lupia A., Mabry P., Madon T., Malhotra N., Mayo-Wilson E., McNutt M., Miguel E., Paluck E. L., Simonsohn U., Soderberg C., Spellman B. A., Turitto J., VandenBos G., Vazire S., Wagenmakers E. J., Wilson R., Yarkoni T., Promoting an open research culture, 10.1126/science.aab2374
  75. Nosek Brian A., Ebersole Charles R., DeHaven Alexander C., Mellor David T., The preregistration revolution, 10.1073/pnas.1708274114
  76. Nosek Brian A., Lakens Daniël, Registered Reports : A Method to Increase the Credibility of Published Results, 10.1027/1864-9335/a000192
  77. Olatunji Bunmi O., Williams Nathan L., Tolin David F., Abramowitz Jonathan S., Sawchuk Craig N., Lohr Jeffrey M., Elwood Lisa S., The Disgust Scale: Item analysis, factor structure, and suggestions for refinement., 10.1037/1040-3590.19.3.281
  78. , Estimating the reproducibility of psychological science, 10.1126/science.aac4716
  79. Oppenheimer Daniel M., Meyvis Tom, Davidenko Nicolas, Instructional manipulation checks: Detecting satisficing to increase statistical power, 10.1016/j.jesp.2009.03.009
  80. Oswald A. J., Wu S., Objective Confirmation of Subjective Measures of Human Well-Being: Evidence from the U.S.A., 10.1126/science.1180606
  81. Risen Jane L., Gilovich Thomas, Why people are reluctant to tempt fate., 10.1037/0022-3514.95.2.293
  82. Robins Richard W., Hendin Holly M., Trzesniewski Kali H., Measuring Global Self-Esteem: Construct Validation of a Single-Item Measure and the Rosenberg Self-Esteem Scale, 10.1177/0146167201272002
  83. Rojas Stephanie L., Widiger Thomas A., Convergent and Discriminant Validity of the Five Factor Form, 10.1177/1073191113517260
  84. Rosenberg Morris, Society and the Adolescent Self-Image, ISBN:9781400876136, 10.1515/9781400876136
  85. Rosenthal Robert, The file drawer problem and tolerance for null results., 10.1037/0033-2909.86.3.638
  86. Ross Lee, Greene David, House Pamela, The “false consensus effect”: An egocentric bias in social perception and attribution processes, 10.1016/0022-1031(77)90049-x
  87. Ross L., The person and the situation: Perspectives of social psychology (1991)
  88. Rottenstreich Yuval, Hsee Christopher K., Money, Kisses, and Electric Shocks: On the Affective Psychology of Risk, 10.1111/1467-9280.00334
  89. Rücker Gerta, Schwarzer Guido, Carpenter James R, Schumacher Martin, Undue reliance on I 2 in assessing heterogeneity may mislead, 10.1186/1471-2288-8-79
  90. Sandvik Ed, Diener Ed, Seidlitz Larry, Subjective Well-Being: The Convergence and Stability of Self-Report and Non-Self-Report Measures, 10.1111/j.1467-6494.1993.tb00283.x
  91. Savani Krishna, Markus Hazel Rose, Naidu N.V.R., Kumar Satishchandra, Berlia Neha, What Counts as a Choice? : U.S. Americans Are More Likely Than Indians to Construe Actions as Choices, 10.1177/0956797609359908
  92. Schwarz Norbert, Clore Gerald L., Evaluating Psychological Research Requires More Than Attention to the N : A Comment on Simonsohn’s (2015) “Small Telescopes”, 10.1177/0956797616653102
  93. Monin Benoît, Oppenheimer Daniel M., Ferguson Melissa J., Carter Travis J., Hassin Ran R., Crisp Richard J., Miles Eleanor, Husnu Shenel, Schwarz Norbert, Strack Fritz, Klein Richard A., Ratliff Kate A., Vianello Michelangelo, Adams Reginald B., Bahník Štěpán, Bernstein Michael J., Bocian Konrad, Brandt Mark J., Brooks Beach, Chloe Brumbaugh Claudia, Cemalcilar Zeynep, Chandler Jesse, Cheong Winnee, Davis William E., Devos Thierry, Eisner Matthew, Frankowska Natalia, Furrow David, Maria Galliani Elisa, Hasselman Fred, Hicks Joshua A., Hovermale James F., Jane Hunt S., Huntsinger Jeffrey R., IJzerman Hans, John Melissa-Sue, Joy-Gaba Jennifer A., Barry Kappes Heather, Krueger Lacy E., Kurtz Jaime, Levitan Carmel A., Mallett Robyn K., Morris Wendy L., Nelson Anthony J., Nier Jason A., Packard Grant, Pilati Ronaldo, Rutchick Abraham M., Schmidt Kathleen, Skorinko Jeanine L., Smith Robert, Steiner Troy G., Storbeck Justin, Van Swol Lyn M., Thompson Donna, van ‘t Veer A. E., Ann Vaughn Leigh, Vranka Marek, Wichman Aaron L., Woodzicka Julie A., Nosek Brian A., Kahneman Daniel, Commentaries and Rejoinder on, 10.1027/1864-9335/a000202
  94. Schwarz Norbert, Strack Fritz, Mai Hans-Peter, Assimilation and Contrast Effects in Part-Whole Question Sequences: A Conversational Logic Analysis, 10.1086/269239
  95. Shafir Eldar, Choosing versus rejecting: Why some options are both better and worse than others, 10.3758/bf03197186
  96. Simmons Joseph P., Nelson Leif D., Simonsohn Uri, False-Positive Psychology : Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant, 10.1177/0956797611417632
  97. Simonsohn Uri, Small Telescopes : Detectability and the Evaluation of Replication Results, 10.1177/0956797614567341
  98. Srull Thomas K., Wyer Robert S., The role of category accessibility in the interpretation of information about persons: Some determinants and implications., 10.1037/0022-3514.37.10.1660
  99. Stroebe Wolfgang, Strack Fritz, The Alleged Crisis and the Illusion of Exact Replication, 10.1177/1745691613514450
  100. Todd Andrew R., Hanko Karlene, Galinsky Adam D., Mussweiler Thomas, When Focusing on Differences Leads to Similar Perspectives, 10.1177/0956797610392929
  101. Tversky A., Cognition and Categorization, 1, 79 (1978)
  102. Tversky A, Kahneman D, The framing of decisions and the psychology of choice, 10.1126/science.7455683
  103. Van Bavel Jay J., Mende-Siedlecki Peter, Brady William J., Reinero Diego A., Contextual sensitivity in scientific reproducibility, 10.1073/pnas.1521897113
  104. Van Lange Paul A. M., De Bruin Ellen M. N., Otten Wilma, Joireman Jeffrey A., Development of prosocial, individualistic, and competitive orientations: Theory and preliminary evidence., 10.1037/0022-3514.73.4.733
  105. Veenhoven Ruut, The International Scale Interval Study: Improving the Comparability of Responses to Survey Questions About Happiness, Social Indicators Research Series ISBN:9781402085680 p.45-58, 10.1007/978-1-4020-8569-7_5
  106. Viechtbauer Wolfgang, Conducting Meta-Analyses inRwith themetaforPackage, 10.18637/jss.v036.i03
  107. Wagenmakers Eric-Jan, Wetzels Ruud, Borsboom Denny, van der Maas Han L. J., Kievit Rogier A., An Agenda for Purely Confirmatory Research, 10.1177/1745691612463078
  108. Zaval Lisa, Keenan Elizabeth A., Johnson Eric J., Weber Elke U., How warm days increase belief in global warming, 10.1038/nclimate2093
  109. Zhong C.-B., Liljenquist K., Washing Away Your Sins: Threatened Morality and Physical Cleansing, 10.1126/science.1130726
Bibliographic reference Klein, Richard ; Cambier, Fanny ; Kervyn de Meerendré, Nicolas ; et. al. Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. In: Advances in methods and practices in psychological science, Vol. 1, no.4, p. 443-490 (2018)
Permanent URL http://hdl.handle.net/2078.1/214692