READ

Fascinating, but does it replicate?

The replication crisis is undermining scientific trust.
Tom Gurn
Tom Gurn
Freelance Writer
Fascinating, but does it replicate?

Over the last few centuries, the scientific method has established itself as a pretty useful tool

A key facet of scientific methodology is reproducibility. 

Essentially, good science should be easily reproduced with the same methodology in a different setting.

Ten years ago, Nature published an article noting that more than half of psychological studies cannot be replicated.

It seems little has changed since then, with claims that sociology, economics, medicine and other fields are all experiencing reproducibility issues. 

Dodgy dudes doing dodgy things

Science is littered with dodgy work that, despite heavy criticism, remains influential.

Take the Stanford Prison Experiment. A team of researchers randomly assigned volunteers as either prisoners or guards to study behaviour in jails. 

The study was cancelled almost immediately after ‘prisoners’ were abused by ‘guards’.


Caption: The Stanford Prison Experiment was called off after less than a week.
Credit: Philip Zimbardo

This was unreplicable for several reasons. First, ‘prisoners’ were not protected from the ‘guards’, which cannot be allowed to happen again. 

Second, the lead researcher Phillip Zimbardo was instructing the guards about how to act, undermining the experiment’s premise. 

Scientists have nevertheless attempted to replicate something similar with inconsistent results.

Digital disinformation 

Associate Professor Chris Kavanagh is a cultural anthropologist and psychologist at Rikkyo University. 

He has been vocal about this issue, pointing out the pervasiveness of the replication crisis due to unregulated podcasts like The Huberman Lab and popular science YouTube channels such as TED. 

The influential TED Talk Your Body Language May Shape Who You Are is guilty of this.

Social psychologist Dr Amy Cuddy claims that altering your posture can change how you feel and forcing a smile can make you happier. 

She reached this conclusion after measuring cortisol levels in people’s brains after they adopted a ‘power’ pose for just minutes. 

Supposedly, people who posed like a gorilla felt more confident and were less stressed.

The problem is that power poses aren’t so powerful under a rigorous scientific lens. 

Scientists have struggled to replicate Cuddy’s research, but the TED Talk has amassed an impressive 27 million views.

Publish or perish

Chris argues there’s fault on both sides of the academic publishing industry.

“There’s a well-known issue called publication bias,” he says.

Publication bias is the propensity for academic journals to prioritise studies with positive, statistically significant results. 

“Journals want to publish interesting results with novel findings, but science relies on negative results and things which aren’t really groundbreaking,” says Chris. 

“They deserve a lot of the blame for the situation because a lot of the incentives were allowing questionable research practices.”

That said, researchers contribute to the problem as well. 

“Academics are trained to respond to the incentives of the publishing industry,” says Chris.

He says students are told, “You’ve collected all this data and the results are negative, you don’t need to panic.” 

“Go back to your data with an open mind … Control for different things and see if you can find a result.”

Caption: Researchers are sometimes encouraged to try and find a different result.
Credit: Matt Cardy/Getty Images

This kind of retrospective malpractice is known as p-hacking

The p-value is a measure of how likely the observed outcome can be attributed to chance.  

This p-hacking generally occurs when scientists reassess their hypothesis after the data has been collected due to the results being deemed boring or insignificant. 

Just pre better

In recent years, things appear to have improved due to organisations such as the Journal of Negative Results, a now defunct journal dedicated to publishing studies with no significant outcomes. 

Or there’s the Many Labs Project, where 51 researchers from 35 institutions tested the outcomes of significant studies using exactly the same methodology, often with results that the original researchers found less than satisfying. 

Since the replication alarm bells rang a few years ago, a lot of the social sciences have cleaned up their act. 

Chris suggests this is thanks to an increase in the number of pre-registered studies.

Pre-registration is the process of recording the methods and analysis of your study into an open-source database before the study has been conducted. 

That way, you can’t skew the data afterwards to suit preconceived notions or biases from academic publishers

However, pre-registration isn’t a silver bullet.

“Very few people go and check pre-registrations. Very few people download datasets and analyse them,” says Chris. 

“It’s not usually a prerequisite for publication.” 

So while we have good reason to trust in the scientific process of criticism and debate, there are still plenty of gaps for crappy science to slip through.

Tom Gurn
About the author
Tom Gurn
Tom Gurn is a freelance writer from Kaurna Yerta (Adelaide), South Australia
View articles
Tom Gurn is a freelance writer from Kaurna Yerta (Adelaide), South Australia
View articles

NEXT ARTICLE

We've got chemistry, let's take it to the next level!

Get the latest WA science news delivered to your inbox, every fortnight.

This field is for validation purposes and should be left unchanged.

Republish

Creative Commons Logo

Republishing our content

We want our stories to be shared and seen by as many people as possible.

Therefore, unless it says otherwise, copyright on the stories on Particle belongs to Scitech and they are published under a Creative Commons Attribution-NoDerivatives 4.0 International License.

This allows you to republish our articles online or in print for free. You just need to credit us and link to us, and you can’t edit our material or sell it separately.

Using the ‘republish’ button on our website is the easiest way to meet our guidelines.

Guidelines

You cannot edit the article.

When republishing, you have to credit our authors, ideally in the byline. You have to credit Particle with a link back to the original publication on Particle.

If you’re republishing online, you must use our pageview counter, link to us and include links from our story. Our page view counter is a small pixel-ping (invisible to the eye) that allows us to know when our content is republished. It’s a condition of our guidelines that you include our counter. If you use the ‘republish’ then you’ll capture our page counter.

If you’re republishing in print, please email us to let us so we know about it (we get very proud to see our work republished) and you must include the Particle logo next to the credits. Download logo here.

If you wish to republish all our stories, please contact us directly to discuss this opportunity.

Images

Most of the images used on Particle are copyright of the photographer who made them.

It is your responsibility to confirm that you’re licensed to republish images in our articles.

Video

All Particle videos can be accessed through YouTube under the Standard YouTube Licence.

The Standard YouTube licence

  1. This licence is ‘All Rights Reserved’, granting provisions for YouTube to display the content, and YouTube’s visitors to stream the content. This means that the content may be streamed from YouTube but specifically forbids downloading, adaptation, and redistribution, except where otherwise licensed. When uploading your content to YouTube it will automatically use the Standard YouTube licence. You can check this by clicking on Advanced Settings and looking at the dropdown box ‘License and rights ownership’.
  2. When a user is uploading a video he has license options that he can choose from. The first option is “standard YouTube License” which means that you grant the broadcasting rights to YouTube. This essentially means that your video can only be accessed from YouTube for watching purpose and cannot be reproduced or distributed in any other form without your consent.

Contact

For more information about using our content, email us: particle@scitech.org.au

Copy this HTML into your CMS
Press Ctrl+C to copy