Search This Blog

Loading...

Sunday, April 19, 2015

Learning How to Fail

I have been taking stock of my 50 years since I left Wichita. How I have existed fills me with horror, for I failed at everything. Spelling, arithmetic, writing, swimming, tennis, golf, dancing, singing, acting, wife, mistress, whore, friend... even cooking. And I do not excuse myself with the usual escape of not trying. I tried with all my heart. - Louis Brooks

I haven't failed. I've just found 1000 ways that don't work. - Thomas Edison

Failing is subjective. The problem with the average judgement of success and failure is that the outcomes are valued more than the effort (the trying). If there was formal instruction on how to conduct research, there would need to be a semester or two on failure. What is it and how do we deal with it? When is it good and when is it bad? How can we write about failure in ways that will be helpful to others?

How can we truly judge an effort (the trying) without the bias that comes from knowing the outcome?

When I was a Boy Scout we had a meeting with about 30 of us boys. The Scout Master told us we were going to have a contest to see who could hold their breath the longest. I took in a deep breath and held my breath for as long as I possibly could, about 50 seconds. The winner clocked in around 4 minutes. This was, of course, a lesson in Boy Scouts never telling lies. When it was over the grown ups had a good laugh. I forget what the joke was, but the Scout Master made his point. No one can hold their breath for that long. In order to achieve the outcome we wanted (winning a prize for being #1) we would have to cheat. The only question the Scout Master needed answered was how long we would cheat.

In my class on coping with perceived research failure, I would start with Feynmans Cargo Cult speech.

But there is one feature I notice that is generally missing in cargo cult science. That is the idea that we all hope you have learned in studying science in school--we never explicitly say what this is, but just hope that you catch on by all the examples of scientific investigation.

We can't hope that the kids will catch on to the ways of scientific investigation. We can face this dilemna head on. In order to teach lessons on failure we must define what it is. How do we distinguish between a failure, such as using the wrong reagents, versus failing to get the results we expect? How can we teach a student how to look at failure the way Thomas Edison did?

Having worked with Cargo Cults so often I can design several laboratory experiments to test the students in a similar manner to my Boy Scout leader. For example, phage display using New England Biolab kits. See how the students treat the repeat sequences and/or contiminants. Test how the students think about the results. There are papers that have been published that tell us what are the most probable explanations for certain sequences are. Will the students find those papers in their attempts to get to the truth? Will the students make the same mistakes that others have made in their attempts to move the research along to more interesting avenues of creating a narrative.

We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory.

And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.

Wednesday, February 25, 2015

The Narrative

It has always seemed strange to me... the things we admire in men, kindness and generosity, openness, honesty, understanding and feeling, are the concomitants of failure in our system. And those traits we detest, sharpness, greed, acquisitiveness, meanness, egotism and self-interest, are the traits of success. And while men admire the quality of the first, they love the produce of the second. - John Steinbeck

The goal of the politician is to communicate convincingly that s/he operates with the quality of the first to achieve the produce of the second. Science is no different. Our leaders are in powerful positions. They get to tell the story of how they are a part of the productive world of science. How honest are they really?

Sinatra

When you live a life like Frank Sinatra you leave behind plenty of information about who you were. Movies, records, interviews, books, and photographs are there for anyone to tell Franks story. There is one piece film footage that I've seen on two separate documentaries. In the first documentary you see Sinatra surrounded by ten burly men. They are rehearsing for a scene for a movie while Frank was taking a break from recording music in a Capital Records studio. The gist of this scene in the documentary is that Frank is a consumate workaholic. In the next documentary you have the same film footage. It starts just when the rehearsing begins, giving the idea that this is not acting but a moment in his actual life. The scene Frank and friends are going over is one where Frank is playing a tough guy. He calls one of the other men out while the rest stand back in deference to the boss. The gist is that Frank really is a tough guy.

Compare a couple of harmless Sinatra retrospectives to the narrative of a scientists work. In spite of the seemingly objective structure of scientific publications, the fact remains that a publication is merely a narrative. How many hours of work are being represented? How many months? Years? Who did what and how many times did they do it? A carefully edited bit of reality will quickly turn the truth to bullshit. Thus, the narrative is worthy of study as well as the thesis being supported.

Narrative

noun
1. a story or account of events, experiences, or the like, whether true or fictitious.
2. a book, literary work, etc., containing such a story.
3. the art, technique, or process of narrating, or of telling a story:

What scientists want everyone to believe is that we follow a superior set of rules that guide each and every one of us to a superior understanding of our world. What a study of "the narrative" put forth by scientists might uncover are, as Steinbeck called the concomitants of failure and the traits of our success.

Is it possible that "The Narrative" lies at the heart of our problems? Could a change in how we communicate deliver a blow to the cargo cult careerists among us? Can we design a better way, a solution, to share the experience of our work and what we think it means?

Once again, ladies and gentlemen, Betrayers of the Truth - William Broad and Nicholas Wade

Considered as a literary form, a scientific paper is as stylized as a sonnet; if it fails to obey rigid rules of composition, it will simply not be published. In essence, the rules require that an experiment be reported as if every aspect of the procedure had been performed according to the philosophers' prescriptions. The conventions of scientific reporting require the writer to be totally impersonal, so as to give the appearance of objectivity.

One solution is to replace the peer review system with a system where data and notebooks are actually reviewed by scientists trained in dissecting "The Narrative". This would create a new career path for scientists. Imagine also a private firm that performs due diligence for investors and/or charities. Is it possible to propose such an affront to the status quo of peer reviewed narratives?


Monday, February 23, 2015

Chapter One

The History of the Narrative

A Cargo Cult Tribesman Discusses His Life In Biotechnology

The company was situated in one story strip mall style industrial area in Woodland Hills California. The employee parking lot was stuffed in the center of the buildings. Wind did not stir the air in our secluded parking lot. It was quite. By 5 o'clock it was always empty. I walked to my car at the days end in this uncomfortable place. I hadn't done anything useful today, I thought. My mind seemed to be as empty and quite. No profound thoughts had stirred my brainwaves. It was as though I had been in REM for hours. No matter what I did I couldn't get the technology to work as promised. I couldn't prove that the guys working on selecting drug targets were correct. I couldn't get the RNA interference, the latest big thing in biotechnology, to work as promised. Was it me?

Steve Jobs said that the most important thing is the project. Great things in business are not done by one person, they are done by a team of people. You hire an A Team and you get the job done. You will encounter people who are difficult to manage. They are passionate about what they do. There will be fights. There will be a lot of noise coming out of that conference room from time to time. We all seemed to get along just fine. Some nights however, there was a little noise to be heard in the parking lot. We would occasionally open the loading dock door and have a noon time barbeque. We would have ribs, hamburgers, potato salad, chips, soda... and beer. Lots of beer. The beer drinkers knew the end was certain so they were going to enjoy the final days. I was no prude. I would have a couple beers at lunch but I had a long drive home. I'd go back to work. It made me nervous to think this was even allowed. At five o'clock I'd make that lonely journey out to my car. The BBQ would have degenerated into a wasted day of drinking beer in a sad lonely parking lot. The A team.

When the end finally came, nine months after I started, three years after the company began, it was just as quiet inside as it was out back on a non-drinking day. I entered through the lab. Empty. I checked the front desk. Hmm. No one here yet. I was early as usual. No one in the cubicle area. Finally, someone appeared. It was the Chief Scientific Officer coming out of his office. "Come on in Sal". We were out of money. It was over and he meant today. "Go on home and start your job search. Use me as a reference. You did a good job."

Monday, February 16, 2015

The Cycle and The Betrayers of the Truth

When The Great Recession hit in 2008, Jamie Diamon of JP Morgan said, "My daughter called me from school one day, and said, 'Dad, what's a financial crisis?' And, without trying to be funny, I said, 'This type of thing happens every five to seven years.'"

That was 7 years ago. Are we due for another "this type of thing"? If so, how will the cargo cults of biotech fare after their surge in IPOs?

A cargo cult does not get to experience the arrival of cargo. It doesn't come once every 5 to 7 years. They spend their lives performing ceremonies in hopes of one day enjoying the cargo. Likewise, biotechnology and the pharmaceutical industry employees/scientists can spend their whole careers in hopes of one day working on a successful drug project. It may never come. There are no cycles in real science.

Western investors are just as susceptible to the lure of ceremony as any cargo cult tribesman. They listen to scientists who have the best credentials. They invest in companies that are using the latest Nobel Prize winning "science". They look for companies that need money within a certain range, as opposed to a specific amount for specific tasks. They tend to differ, however, in the attitude towards the lack of cargo. They won't wait a lifetime to see results. Thus, we should be able to predict our that investment cargo cults will begin to pull out of their biotech cargo cult investments.

What makes investment even more risky is the system of conducting science in modern times. It is not as the professional community of scientists would have you believe. We are led to believe that science is a strictly logical process, involving skilled objective scientists performing experiments in the laboratory, that are rigorously verified by other scientists. All work is published and shared in order to advance the basic science. Any deviations from the truth are weeded out in a self-policing manner.

In the 1982 book, "Betrayers of the Truth, Fraud and Deceit in the Halls of Science" by William Broad and Nicholas Wade, a different description of how science is conducted was offered. "Logic, replication, peer review, objectivity - all had been successfully defied the scientific forgers, often for extended periods of time." This book written right around the advent of our modern biotechnology industry, did not think about the influence of a multi-billion dollar industry. It focused only on the problems of science.

Chapters in Betrayers of the Truth include; 1)The Flawed Ideal 2) Deceit in History 3) Rise of the Careerists 4) The Limits of Replication 5) Power of the Elite 6) Self-Deception and Gullibility 7)Myth of Logic 8) Masters and Apprentices 9) Immunity of Scrutiny 10) Retreat Under Pressure 11) The Failure of Objectivity 12)Fraud and the Structure of Science. In 1982 the biopharma world began spending billions and billions of dollars resulting in a handful of millionaires and no significant medical advances. William Broad and Nicholas Wade had sent out a warning. As they explained

Fraud, we believe, offers another route to understanding science. medicine, after all, has derived much useful knowledge about the normal functioning of the body from the study of its pathology. By studying science through its pathology rather than through some preconceived criterion, it is easier to see the process as it is , as distinct from how it ought to be.

The pathology of the life sciences is like a big juicy piece of fruit ready to be picked from the vine. Has anything changed since the publication of Betrayers of the Truth in 1982? Have we taken a few hundred billion dollars and amplified the problems of careerism and non-reproducible work? Clearly fraud continues to be a part of everyday life in the sciences. We are no better at preventing nor spotting deceit. The cargo cult aspects of science and the biotech industry remain steady.

There is a pattern however. The layoffs have begun. Sanofi, InterMune, Abbott, Syngenta AG, GSK, and Amgen all have plans to cut their workforce this year. All signs point towards a rough 2015. The consequences of "Fraud and the Structure of Science" do have a pattern.

---------------------------------------

In the documentary "I Am" movie director/producer Tom Shadyac asked two questions, "What is wrong with the world and what can we do to fix it?" I have long been asking the same questions about the field in which I am passionate about and in which I once worked. What is wrong is complex. I have spent the last year writing a summary of ideas from this blog that very closely mirror "Betrayers of the Truth". Many of the themes, such as careerism, the structure of peer review and publish or parish, and the simple notion that science is a human endeavor thus subject to constant fraud and deceit, are explored. It is my hope to offer solutions for useful careers in science, not the cycle of the money. I will spend the next year or two or three... editing and self-doubting the effort. Then e-publish and offer my thoughts on this blog.

Sunday, January 18, 2015

Anil Potti vs Bradford Perez

"One day everything will be well, that is our hope. Everything's fine today, that is our illusion" - Voltaire



The Anil Potti saga is not just an isolated case of scientific misconduct. It is a case that follows the known patterns of cargo cult science. The latest news from the Duke University Potti scandal adds a few more details.

Let's first review the patterns of cargo cult science.

Step one: A narrative capable of being published is formulated by a person with above average ambition.

Step two: A skeleton outline of experimental design is set on stone by the ambitious author of the narrative. The design must protect the narrative. Assign a subservient laboratory work force to conduct the experiments.

Step three: The misconduct occurs in the analysis of the data. The author of the narrative aligns the data to the narrative. Any egregious deviations are dealt with by "correcting" the work of the subservient laboratory staff.

Step four: The narrative is written up and submitted to the journals.

In the Anil Potti case:

Step One: Anil Potti, a Principal Investigator at a prominent University, formulates a narrative, popular among the cargo cults. Genetic markers can be identified to help western medical professionals treat disease. Everyone loves a winner, and Anil Potti knew how to convince others that he was a winner. Anil Potti was not the only person with above average ambition. His superiors and most of the people around him supported what he was doing because his narrative was so attractive.

Step Two: Bradford Perez, a third year med student, is assigned to carry out the work that supported Anil Pottis' narrative. He came to the realization that the methods being used to assign patients to clinical trials were not validated. I.E. the methods were more narrative than scientific method. Yet Bradford Perez was in no position to influence the direction of Anil Pottis research/career. As Arthur Caplan, director of the Division of Medical Ethics at the NYU Langone Med Center who is reviewing this case, said, "I have a feeling his lowly status made him someone that they would be able to hope would just go away. There was a little bit of don't-let-the-door-hit-you-on-the-way-out."

Step Three: Perez made several attempts to discuss the methodological issues with Potti. Things do not go well for Perez. He has to make a decision; speak truth to power or join the power in the continuation of the Potti narrative. Validation techniques from the Potti narrative amounted to, "erasing the samples that don't fit the cross validation from the figure and then reporting the cross validation as meaningful and justification for a good predictor".

Step Four: Rather than publishing another paper supporting the Potti narrative, Bradford Perez decides to pursue the courageous path. He writes a 3 page single spaced summary of his concerns with Pottis lab. "I have nothing to gain and much to lose", stated Perez in his letter to the University. He gave up the opportunity to be included as an author on at least 4 manuscripts, a Merit Award for poster proetnation at the ASCO meeting, and a year of med school. He had to put in another year to replace the dishonest research with something in which he could take pride.

The politics of this case are worth studying. Not on a simple blog but at the level of real leadership. As we can see from this case however, the leadership is a large part of the problem. Anil Potti is indeed a Cargo Cult Hall of Famer, but what about everyone else around him? Below and above, many people were spending their scientific careers standing next to the steaming pile of cargo cult science that Anil Potti was putting forth. Why was Bradford Perez the only insider speaking out? Why did his words go unheard by so many for so long?

The rest of us must operate, as Bradford Perez did, against this powerful non-scientific political force. The Cargo Cults are not simple fiefdoms rang by rogue PIs. Rather the Anil Potti story is one of bullying by a powerful person who has no reverence for the scientific method. Political prowess continues to butt heads with the truth. The truth will always win in the long run. But how long is that run and who do we encounter along the way?



Wednesday, December 17, 2014

How Can You Not?

http://www.opb.org/news/article/npr-scientists-often-skip-a-simple-test-that-could-verify-their-work/

This is a clear case of a Cargo Cult leadership. Quality control/assurance is not possible at the NIH? The elephant in the room is how so many researchers get the results they seek regardless of what cell lines the use.

Why I Write

Monday, September 01, 2014

Optimism Is Essential For Our Success?

“I have always believed that scientific research is another domain where a form of optimism is essential to success: I have yet to meet a successful scientist who lacks the ability to exaggerate the importance of what he or she is doing, and I believe that someone who lacks a delusional sense of significance will wilt in the face of repeated experiences of multiple small failures and rare successes, the fate of most researchers.” ― Daniel Kahneman, Thinking, Fast and Slow
This is probably the most profound quote from 'Thinking Fast and Slow', pertaining to the CCS. However much I disagree with the concept of optimism, a human emotion, being essential for success in science, the statement is true. The fate of most researchers is to wilt in the face of repeated experiences of multiple small failures and rare successes. We get beat down from our inability to impress with our research. Just ask the 660 Amgen employees in the Seattle area who are facing an unknown future. Our version of the scientific method does not seem to work as well as other scientists. In order to succeed in a science career, we have to appear optimistic. I, the CCS, am not an optimist. I believe in the power of the negative. If I put my hand on a red hot skillet, I accept the negative burst of pain that prompts me to relocate my hand to prevent permanent damage. Negative outcomes are merely perceptions of the viewer. Any outcome is a clue for the researcher. You can choose to accept what you see or you can try again hoping for a different outcome. Success occurs when you accurately explain an outcome.
That is the idea that we all hope you have learned in studying science in school--we never say explicitly what this is, but just hope that you catch on by all the examples of scientific investigation. It is interesting, therefore, to bring it out now and speak of it explicitly. It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated.
The person who can do this and not suffer the consequences of seeming less than optimistic is having a successful science career. The scientist who feels pressure to exaggerate the importance of their work is not as successful as they let on. Lysenko, for example, was a highly successful scientist, if ones rank in the hierarchy in which you serve is the criteria for success. Dipak Das and Diederik Stapel are examples of college professors who have a history of successfully getting their work published. Yet history has to paint a picture of unsuccessful scientists. Their work is not useful. So what is success? Does it require optimism? Certainly Lysenko, Das, and Stapel knew well the politics of a success career. If they could have been better researchers, they would have used the truths they uncovered to accomplish the same success they achieved through deception. Any BS'er will tell you, the truth is a better weapon than a falsehood. The truth does not have to be guarded and kept behind the curtain of Oz. Any time an exaggeration is used, the scientist must be careful not to cross that line Das and Stapel crossed.
“A reliable way of making people believe in falsehoods is frequent repetition, because familiarity is not easily distinguished from truth.” ― Daniel Kahneman, Thinking, Fast and Slow
So sayeth Dr. Kahneman.
We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.
So sayeth Dr. Feynman. A successful science is the product of many individuals who work under the title of "scientist". A "successful scientist" is one of many such individuals who may be contributing positive or negative things to the science. And the truth will come out. A truly successful scientist is one who gains a good reputation in the long run. Our ideas and theories are bigger than our careers, in the long run. What determines a successful scientist is the science behind the rhetoric.