Search This Blog

Thursday, November 01, 2012

Who Wants It

I believe that future generations will look back on us as Feynman looked back to the middle ages:
During the Middle Ages there were all kinds of crazy ideas, such as that a piece of rhinoceros horn would increase potency. Then a method was discovered for separating the ideas--which was to try one to see if it worked, and if it didn't work, to eliminate it. This method became organized, of course, into science.
The idea of reproducing someone elses experiments is nothing new. What is new is the idea that such a project would create so much dissent. A piece of rhino horn used for erectile disfunction could be put to the test in a Viagra style study these days. It would be put to the test if some proponent of its use had the clout to get the research put in motion. Therein is the pickle we find ourselves in with reproducibility. The people with the clout are the ones whose work needs to be subjected to reproducibility studies.
Now when I talk about the actual Reproducibility Initiative I am using it as an example of one such effort to get the conversation started about what actaully takes place in the lives and careers of any modern day participant in the scientific world. I think any reproducible science, even boring science that doesn't rise to the sexy standards narrative science, is better than the cargo cult science. You can get rich, you can get published, but that is not the primary concern with this conversation. What I am thinking about is a Utopian world where the members of the cargo cult begin to be skeptics. They lose their jobs in the watch towers because, as their leaders informed them, they were not wearing the coconut antennas properly. They begin to question the antennas. They build confidence in their own ability to distinguish bullshit from the pile of clever sciencey narratives that their leaders pile on them. They discover a method for separating the ideas--to try one and see if it worked, and if it didn't work, to eliminate it.

Having started with this concept, that science began not so long ago with the idea of testing popular ideas, I'll address the comment from my last post. (Thanks for the comment btw)

As long as the incentives are not directed towards reproducibility, it won't be happen. The Amgen study was great but we didn't hear about until years after the original work and the authors had left the company and felt free to publish.
And of course we still don't know which studies or which labs were involved due to confidentiality agreements.
I'm not sure what is meant here. The incentives in reproducible science is that it leads to bigger and better things. We see so far because we stand on the shoulders of giants.

For academics funding is limiting and precious. If you tell an academic "here is $10,000 and you can spend it on reproducing your own work or you can spend it on a new project which will bring you more money and more publications", which will they choose?
Who would be so foolish as to give someone this choice? I've never worked with anyone who would volunteer for an outside reproducibility study of their own work. As I said at the end of my last post however, we have to be creative. Step out of the mindset of an academic scientist. Think in terms of a beaurocrat who decides on the NIH budget or an investor who wants to give biotech one more chance. You have a choice: Fund the same old guys who burned you back in the 90s and 2000s, or tap into a new paradigm. Only fund people who agree to have their work double checked. We're all only human. That is what the same old guys don't want you to know. Don't believe them. If they don't want the money, find someone else.
If you ask academics, all of them will insist to you that their work is reproducible so why would any of them consider trying to reproduce it?
Again, the academics are not the consumers (yet). They are the ones in whom we've lost faith. A Reproducibility Initiative must first do business with the consumers of good and bad science. The consumers can continue to lose money like they have in the past or they can try and see what certifiably reproducible work can do for them.

tldr: You don't get into Science and Nature by reproducing published research.
Again, no one is trying to get into Science and Nature here. They are trying to get Science and Nature to get into reproducible science.
Thanks to whomever this commenter was. You are absolutely correct with everything you said. The issues you bring up are the exact issues to attack. The question is how. Can we find a stunt like the Carnegie elephant crossing a steel bridge stunt? We have to be creative. The academic scientist is not the guy who wants a reproducibility group hanging over their research. In time however, the people with the money will come to find investments in reproducible science to be far more lucrative than the investments they made without this safety net. Soon scientists will have no choice, if they want funding, but to conduct the kind of work that will be subjected to reproducibility.

I know I'm only dreaming. But this is all just a thought experiment.

1 comment:

Anonymous said...

Some good points in your response.

I'm the same individual that left that comment.

The incentives in reproducible science is that it leads to bigger and better things.

That's true but the problem from the standpoint of the producers and consumers of science is that the process of generating "bigger and better things" isn't rapid enough for them which leads science producers to over promise and under-deliver on a routine basis. Why the consumers of science (investors and government folks) put up with this situation is not something that I've thought too much about or know the answer to. Maybe it's fear and desperation. Maybe it's the same reason people stay in abusive relationships.

Can we find a stunt like the Carnegie elephant crossing a steel bridge stunt?

I think that all it would require to shift the entire scientific community is to make an example of a few of these PIs publishing irreproducible science. If some organisation with resources were to pick one or two papers from every issue of Nature or Cell and attempt to reproduce them and make those results public it would have a tremendous impact on everyone else. After a couple of high profile retractions, scientists would be falling all over themselves to make sure that their work is reproducible before they submit to journals.

Once PIs realize that their work has a chance of being subjected to this kind of public scrutiny, they will think twice about the quality of what they submit.

It would be something like Comparative Effectiveness Research. Or you can think of it as a random scientific audit.

The problem of how to get a group of people to do something they wouldn't otherwise do is a very active and well studied area of research in Sociology, Economics, Psychology etc.

For example, the way to make sure people pay taxes is to keep the tax rate low so people don't feel the burden is too great, make it as easy as possible to pay by withholding wages and put in a penalty for fraudulent returns. You need carrots and sticks. Focusing on single approach will be insufficient.

I think just bringing attention to the problem, which your blog does, is an important step.