Cargo Cult Cancer

In the Cargo Cults, authority is the truth. In real science the truth the authority. The documentary Money and Medicine highlighted many examples of our medical authorities working in cargo cult mode and real science. The authority becomes the truth when we just don't know something. We don't know any way of dealing with Alzheimer's but we have doctors who still have to treat patients. They even have pills approved by the FDA. That seems to make people feel better, just as a religion. Science, on the other hand, offers up what some would consider confusion. By admitting that we do not know something we run the risk of losing our credibility with the masses. The minority, those who think scientifically, run towards the unknown with a much different attitude. The truth is our authority. If we can reign it in we can harness some of that authority/power.

In the Money and Medicine documentary the authority take a real science view of what a hospital can and cannot do for end of life situations. We can keep people alive, for example, but in ways that do not bring them back to any semblance of a healthy human being. Death is an option that most would prefer. Yet a hospital must offer the option of keeping a patient alive if they can, even when it is merely a technical version of life. To the healthy person looking in from the outside, life is life. They imagine the dying individual being happy and content to lie in their bed looking out from the inside. The healthy person does not imagine what it is like on the inside, laying in bed as your body tries to expire. We simply don't want to accept the fate that awaits us all.

Cancer may very well await us all, if we live long enough.  If we were to study cancer, and not just single genes or other single causes, we could study dormant cancers we all live with. What cancer precursors dwell within our bodies that never progress to the point of taking our lives. In Money and Medicine was pointed out that autopsies on a group of men, post 70 years of age who died of something other than prostate cancer, showed that around 80% had what doctors would call prostate cancer. The advent of the prostate specific antigen (PSA) test has led to an increase in diagnosis yet dubious evidence of having an impact on mortality. The same can be said for mammography and breast cancer. Men and women have separate but equal cancer precursors that may exist as one of many possible expiration devices placed in our bodies to ensure we don't live too long. When it comes to medical science however, the authority can fancy itself to be the truth. They will find these ticking time bombs and zap them with medicine. As we can see with breast and prostate cancer testing, early detection has not provided the doctors with the authority to hand over new leases on life. They do however offer up treatments that enrich big pharma and the hospitals.

We are humans and humans seek patterns. If we can find patterns in those who die from cancers we believe we can prevent cancer from happening. The news of Angelina Jolies double mastectomy is an extreme example of using the patterns we think we see and following up with another pattern we think we see, that of surgery leading to extended life. Angelina Jolies mother died in her fifties thus Angelina is taking a preventative measure to avoid the same fate. The medical authority has given her a new lease on life, but no guarantees that it makes a difference. We can see how the Hippocratic Oath, "I will prescribe regimens for the good of my patients according to my ability and my judgment and never do harm to anyone", is subjective. 

This all leads me to an article published by the New York Times last month.  Peggy Orenstein wrote this piece, "Our Feel-Good War on Breast Cancer". She points out that early detection may not provide the benefits that have been touted. Younger women have more density in the breasts thus there are more false positives and false negatives in mammograms.  The question of just how many lives are being saved is itself taboo among breast cancer advocates. Their answer is that early detection works. They have what they want, never mind the statistics. It feels good to believe that what we are doing is working. So why the inconvenient negative data?

From the documentary Money and Medicine:

A number of recent studies estimate that a third of all health care expenditures are unnecessary and that eliminating wasteful spending would save over 800 billion dollars a year. 

Why is that? I continue to think about these things. What have I come up with here today? What is my point? Conclusions? After 7 years on this blog I see only one solution. Keep the negative and inconvenient information flowing. To a true scientific mind, the state of not-knowing is like the state of hunger to a person walking into a restaurant filled with the aroma of delicious food. To the cargo cult scientist, the state of not-knowing is shameful. The cult must turn to the authority who will make the claim that they do know something, it's just that a little more research is needed. Research becomes mere tweaking the old ways that aren't working. My conclusion today is to simply stay thirsty my friends. Remain a skeptic. The state of not-knowing is good. Authority does not represent the truth... the truth is the authority.

Creativity

The headlines from the featured stories on Biospace struck me as an interesting study for future generations. Today we can look back and scoff of things like leeching or the uproar over whether to wash ones hands before surgery or not. But look at what our scientists are proposing just today:

• Drinking Champagne Improves Memory
• Could Eating Peppers Prevent Parkinsons?
• Soy and Tomato may be effective in preventing Prostate Cancer
• Fish Oil doesn't help prevent heart attacks

In the future, people might look at this sort of research as we now look at leeching. Each solution to a human condition has a single simple answer. Leeching was suppose to help balance the four humors; blood, phlegm, yellow and black vile. When all humors were in balance perfect health was assured. What if you are experiencing memory loss and the leeches aren't working? Never fear! Science has found an alcoholic beverage to cure that problem. Throw in some red wine, the magical elixir of life, and you might just wake up with a hangover and a newfound skepticism towards this kind of science. 

Each single simple solution began life... as a single simple solution. We are taught this manner of thinking in our education system. Life sciences are easy on the math, heavy on the narrative. As a result it seems that we begin to address problems, such as how memory works, Parkinsons, Prostate Cancer and cardiovascular disease, as if they were test questions. Parkinsons can be treated with ________. Eating peppers can be used to treat __________. We find the answers from simple everyday products. 

Scientifically speaking, the most interesting aspect of this kind of research would be that scientists have found a way to measure disease states. The assumption is that we naturally know how to study health effects one food at a time. At the same time we still have no standardized way of measuring the health benefits of any intervention. If we did clinical trials would be much easier. We could simply fill in the blanks with data from our statistical analysis with an appendix reference to the raw data. The above studies could be compared to other claims on their disease area. Red wine benefits could be compared to the secret of long life (bacon) discovered by this 105 year old Texas woman.

In each case above we have a lack of creativity when it comes to attaching solutions to problems. We have an abundance of creativity in creating the story that the simple solutions actually effect the complex problem. Science should work in the opposite direction. We get creative in tackling problems and rigorously scientific in analyzing our outcomes. 

Much of our problems come from education. This TED talk describes how we have a one-size-fits-all education system that mines each brain for the same things. 

The creativity we employ is not what it once was. Don't tell us about your findings of X treating or curing Y. Tell us how you found a way of measuring the effects of X on Y. That is where creativity is needed. 

Forks Over Knives Revisited

I have, in the past, touted the miracle "cure-all" medicine of a mostly plant based diet. The documentary Forks Over Knives struck me as a wonderful example of people giving up traditional pill based sick care for "let food be thy medicine" health care. They were careful not to use the word vegan in the documentary, but "Forks Over Knives" is about being a vegan. I have looked into many dietary ideologies and there are many many paths one can take that are better than the typical American diet. The Forks Over Knives path isn't bad but tough for those who enjoy a little more fat. Another path could be the Atkins Diet approach. Gary Taubes produced a very detailed documentation of the history of cholesterol research in his book "Good Calories Bad Calories". What struck me about the two paths was how they stood in stark contrast to one another yet achieved similar results. There was one common trait however. The lack of sugar and processed foods.

I came upon this critique of Forks Over Knives the other day. It's a blog written by a 26 year old former vegan who now survives on a raw food diet with some meat in the picture. She is also an English major taking on the MDs who started the Forks Over Knives movement. Her name is Denise Minger. She is Davy fighting Goliath.

We all share a common belief in the powers of what we eat. We share an interest in seeing people get healthy by taking aim at their behavior. We also have skepticism built into our DNA. We all like positive results, but here is where we must avoid being Cargo Cult Scientists. What matters is finding the most probable reasons for the outcomes we like so much.

The first thing we at the CCS must address is part of the documentary regarding the Nazi occupation of Norway. The message was that meat rationing led to a rapid reduction in death from cardiovascular disease. What was missing was the fact that the rationing of meat began one year after the reduction of death from cardiovascular disease.

It's a kind of scientific integrity,
a principle of scientific thought that corresponds to a kind of
utter honesty--a kind of leaning over backwards. For example, if
you're doing an experiment, you should report everything that you
think might make it invalid--not only what you think is right about
it: other causes that could possibly explain your results; and
things you thought of that you've eliminated by some other
experiment, and how they worked--to make sure the other fellow can
tell they have been eliminated. - R. Feynman

During the first year [starting in spring of 1940] the rationing included all imported foods, bread, fats, sugar, coffee, cocoa, syrup, and coffee substitute. In the second year [starting in late 1941] all kinds of meat and pork, eggs, milk and dairy products were rationed…

The question now is how this information was dealt with by the Forks Over Knives people. Did they, as Feyman  instructs, "report everything that you think might make it invalid".  I think this information might make vegans cringe. What other causes could explain these results? Sugar? bread? syrup? Fats? Luckily for us, Denise Minger got the attention of T Colin Cambell. This response was a testament to her work. Unfortunately in his response he failed to specifically address the actual scientific studies that were of dubious quality, yet made it into the film for dramatic affects. 

I'd like to highlight the condescending response and some of the Cargo Cultisms therein. The response begins with a classic ad Hominem logical fallacy, a veiled attack on Ms. Mingers qualifications to question the Great Oz's of medical science. 

Kudos to Ms. Minger for having the interest, and taking the time, to do considerable 

analysis, and for describing her findings in readily accessible language. And kudos to 

her for being clear and admitting, right up front, that she is neither a statistician nor an 

epidemiologist, but an English major with a love for writing and an interest in nutrition. 

We need more people with this kind of interest.

Silly rabbit, Tricks are for kids... and science is for doctors. The Cargo Cult message here is that they are doctors and she is only an English major. Not even an English PhD! 

Next up in the Cargo Cult Science department is the appeal that no specific part of their Cargo Cult ceremony brings the cargo. It is only the complicated intermingling of this-that-and-the-other-thing that makes the narrative work. It's the George Bush Jr. cop-out. Sure we lied to get the war started but now that Sadam is dead, it all worked out for the best. Dr. Campbell organized his thusly:

My response can be divided into three parts, mostly addressing her lack of 
proportionality—what’s important and what’s not. 
 
•Misunderstanding our book’s objectives and my research findings 
 
•Excessive reliance on the use of unadjusted correlations in the China database 
 
•Failure to note the broader implications of choosing the right dietary lifestyle 
Let's slow down the straw man argument for a moment and see if Dr. Campbell addresses the above Nazi/Norway dilemma or the following specific question by Ms. Minger:

What we’re interested in is the sentence near the bottom, which the film’s producers apparently didn’t notice: ”In all, 30 rats on the high-protein diet and 12 on the low-protein diet survived for more than a year.”

To which we get the response:

First and foremost, our extensive work on the biochemical fundamentals of the casein 

effect on experimental cancer in laboratory animals (only partly described in our book) 

was prominent because these findings led to my suggestion of fundamental principles 

and concepts that apply to the broader effects of nutrition on cancer development. 

So... what about the damned rats dying?  What about those sugar-free Norwegian vegans? Dr. Campbell says:

The China research project was a cornerstone study, yes, but it was NOT the sole
determinant of my views (as I have repeated, almost ad nauseum in my lectures). In
doing so, and except for a few denigrating remarks on our experimental animal
research, she also ignores the remaining findings that I presented in our book. She
seems not to understand what our laboratory research was showing. Using univariate
correlations mostly without adjustment for confounding factors, qualification of variable
authenticity, and/or biological plausibility can lead to haphazard evidence, subject to the
whims of personal bias. Also, univariate correlations of this type can lead to too memphasis on individual nutrients and foods as potential causes of events.  

Why wasn't their a reference for the denigrating remarks? More importantly, why didn't Dr. Campbnell start advocating a high protein diet for rats hoping to increase their time in the vivarium. Dr. Campbell again wants the reader to keep focused on his narrative and ignore the facts found in his research laboratory. 

The "univariate correlations" conversation is yet another attempt to hide the somewhat dodgy research findings that were used in presenting the Forks Over Knives narrative. The only thing I have to add to this discussion is this. If they aren't significant, why did we hear so much about them? This area right here, where we are suppose to ignore the little studies, yet still believe the narrative they support. really starts to get old. If it doesn't matter, tell us why. Don't bring it up and berate us non-PhD/MDs and English majors for asking you questions. You are suppose to be defending your research, not the insignificance of what you publish. 

I must commend Denise Minger for what she has done. She is Davy fighting the mighty Goliath. I think we can all agree that we care about food as medicine and being healthy. So much passion has Denise for this issue, that she took on a highly popular movement that is in fact a million times better than the diet on which I was raised. I would like to say something nice about Dr. Campbell but I'll end with something he said to Ms. Minger ... which was very Cargo Cult. 

I should conclude by noting the suggestion of the professional epidemiologist, cited 
above, who suggested that ultimately Denise may wish to publish her findings in a peer- reviewed journal but who presently felt strongly that the current version would not be 
accepted. I concur. 
I have never tired of my own narrative, that most of what gets published is pure bullshit. We all know that science journal editors do not practice the scientific method and thus Ms. Minger does not have the proper credentials to get published. We also know that Dr. Campbell does. What he is suggesting here is that, once again, he is a doctor and she is an English major. You win that one Dr. Campbell. You have surprisingly rejected her paper.



She sure did get Dr. Campbells goat though.



In honor of Ms. Minger, I'm going to sprinkle a little goat cheese on my salad tonight.

Biospace Report 2013 Report On Jobs

Without proper bean counting techniques, it is hard to say how many people have come, gone or stayed in the work force that makes up biotechnology/pharmaceutical science. You could first set up a list of career paths. Let's say one person has an associate degree in biotechnology. The next person has a BS in Chemical Engineering. The next person has an MD. The next a PhD in Pharmacology. Make a second list of possible job opportunities and match up the degrees with the jobs. Then make yet another list of jobs and match up the education backgrounds and job experience with the next leg of the journey of a career.

What is the life span of a particular career and how many should be expected to support the companies that currently exist? Should we expect more or less companies in the future?

According to Biospace, the future is looking brighter. "Biopharma recovery continues with unemployment rates falling over three percent," said Chanille Hewett, Biospace Product Manager. Our salary data shows further recovery indicators; since bottoming out in 2010 biotechnology salaries hit a three year high in 2012. Not only are our professionals getting back to work, but their pay is trending upward as well."

How did they ignore the large number of layoffs during the time period of 2010 to 2012? Does their methodology really tell the true story. It would seem that there is a built in bias. A Biospace Product Manager may be under pressure to tell the "best story". Journalism is not what it used to be. According to Xconomy, Biotech was back in 2010, but has been on the decline every since. Over 150,000 jobs were shed from 2009 to 2012. How is the Biospace "recovery" treating the lost 150,000 jobs?

Using Seattle as an example, less jobs are posted on the WBBA website than ever before. Knowing people in the area, I can identify some of the jobs as being mere place holders on the WBBA website to give off the appearance of a viable company. In reality the jobs have sat on the website for months with no effort being made in-house to fill those positions. Out of 17 companies listing jobs on WBBA, nine companies only have 1 job vacancy, four companies have 2 jobs, leaving the remaining four companies offering 3, 7, 10, and 28 jobs. Take out Seattle Genetics and you've got 37 jobs from 16 companies.

Why bring up a non-factor like Seattle? The area was never a viable hub for generating large numbers of jobs. Old job creator like Institute for Systems Biology no longer have life science opportunities. I have seen people searching for "Institute for Systems Biology layoffs" on my blog. None have been reported. Hmm. If you look at jobs on the Amgen site you will see many that have been listed for years. Lazy human resource practices and embarrassment over the lack of forward progress have added to the confusing calculations done by Biospace. If there is a problem reporting the proper numbers, then larger hubs such as San Diego, San Francisco, North Carolina and Boston, might be adding larger amounts of bad data to the calculations being made by Biospace.

Using the scientific method, we can look into the realities of bio/pharma.

My observation is that there have been far too many layoffs to conclude that in 2013 everything is on the  mend. The measurement of recovery hasn't been defined. Imagine the recovery from the past few years being the inverse of the layoffs. Pfizer hires 19,500 people!

So I have to ask a question. I start with several questions. "What are the jobs that were lost?" "What are the new jobs?" In general, "Has the biopharma workforce changed what is does?"

Using the Cargo Cult analogy, "Have the Cargo Cult Airports figured out how to conduct themselves in a manner that puts more wealth into their system?"

By looking at the new jobs, we have a better way of judging the future of the business. Have we redefined how a person is to stand in the watch towers or have we figured out what the Westerners were actually doing in their watch towers?

My hypothesis is that things have not gotten better, in general, that the workforce has lost valuable experience. The Dilbert-esque bosses have kept their positions while the people, whose minds made up the knowledge base of what really needs to get done, have lost their jobs. As the money fades away, careerism overshadows what is truly needed at the company. Those with the power to choose will always choose to keep themselves around and get rid of those below. Proving ones value does not get done via the scientific method.

Check what people are saying online. Biofind Rumor Mill:

Submitted by Anonymous on Thu, 10/01/2013 - 03:12.

We had two openings for entry level (bs +1/2 years experience) molecular bio techs. We received over 100 resumes, over 30 of which were from PhD holders. We hired people with bs + 2-5 years experience. The overdegreed candidates didn't even get to the interview stage.

Submitted by Anonymous on Sun, 29/07/2012 - 03:30.

In the last 5 years, jobs in science/pharma/biotech have declined resulting in a surplus of unemployed PhD and MS scientists. We need to have some organization that will help MS/PhD scientists get jobs in the fields outside of their expertise with some training while working. A person who has done MS/PhD can certainly learn other skills in 1-2 years and succeed in other fields.

If you continue reading the thread from Biofind you will see a trend. Over-education is detrimental to your hiring prospects. Training is not provided quite often. The industry where you hope to have a career is not booming. Some will win, most will not, and the winners aren't training the losers how to succeed.

What is the experiment? Well... I started this blog in 2006. That has been the experiment. Was I able to find a problem other than my own career falling by the wayside? The industry has proven, in spite of reports of its constant rising from the ashes (without any reports of a fire) that it is a poor investment. The money has dried up. Bad science leads to bad investments. The intolerance to the loss of capital is a good thing. New initiatives, such as the Reproducibility Initiative and the latest initiative from the journal Nature to improve the reproducibility of their published papers, are starting to pop up. The Amgen study boldly went where few professional scientists would dare go, and they were published. Retraction Watch now shines a light on things we know... that aren't so. People are starting to do what they have been reluctant to do in the past, admit our faults. Ben Goldacre has published Bad Pharma that has opened the eyes of regulators to the plight of anyone who wants to get the to the truth of a big pharma clinical trial. Aubrey Blumsons war against Procter and Gamble opened the door for questioning big pharma trials and their lack of transparency. Cargo Cult companies like SIRNA and Sirtris have closed up shop, costing their big pharma purchasers huge write-offs. Beyond biotech/pharma, medical practices have also been scrutinized in books like Checklist Manifesto and Unaccountable. 

In general, we are not on the mend as often reported by Biospace and other people who benefit from maintaining the status quo. We are in the process of changing, getting rid of bad science, and a whole lot of bad scientists. The old ways were unsustainable. We are learning to be more honest and hold others to the same standards. The experiment continues.

Euclids Elements

In 'Euclids Window' by Leonard Mlodinow we are taken through the history of geometry. It's not at all boring as some might assume. There is a history lesson on Alexandria, the leaders, and the library for example. It takes the imagination to a time where information was kept precariously on rolled up sheets of papyrus. Alexander the Great borrowed such texts from other leaders and didn't give them back! Imagine how much easier our own lives are now yet how much lazier that has made us. In the days before the 1000 year dark age, geek knowledge was in fashion. The math employed by the Egyptians was a commodity that the Greeks would travel by wooden ships to obtain. This made the information physically valuable to collectors of all things foreign. Think of those individuals as venture capitalists. They know something is valuable by they know not why. It was intellectually valuable to Euclid and the scientists he influenced. They knew why.

Euclids, 'Elements', his seminal work on Geometry, relied on the accumulated information on the subject. It is believed that most of the theorems were proven by other mathematicians. Perhaps the first sign of a powerful new scientific understanding would be a tall building or a way of having water available for farming. Egypt for example, had the pyramids. Someone capable of understanding the science and engineering had to grab some papyrus and go find out how they built them. It took Euclid many years to collect information from these kinds of writings and finally sit down to begin his work. He laid out his lifetime of interest in geometry, the things he felt were powerful in 13 rolls of papyrus.

This collection, 'Euclids Elements' has influenced people in ways beyond geometry. Bertrand Russell, for example, adopted the axiomatized deductive structures that Euclid's work introduced. (I stole "axiomatized deductive structures" from Wikipedia.) It is important to understand that sentence though. Axiom: A starting point of reasoning. Deductive reasoning links premises with conclusions. The structure of Elements is genius. 

Let's apply some "axiomatized deductive structure" to the Cargo Cults of biotechnology. There is much that we have learned since 1980 regarding the life sciences. As far as biotechnology and medical science are concerned however, we have created a huge steaming pile of information. It is not easy to find the starting point of much of the research then follow it logically to the conclusions that have been set in stone. I say "set in stone" because that is how biotechnology science begins. Let's look again at On-Q-ity. 

A group of investors will put money up to advance a science/technology towards a profitable healthcare product or service. On-Q-ity diagnosed cancer using biomarkers. After the money is in place PhDs are hired to instruct bachelor degreed people on how to "prove" the science and technology are useful. The conclusions are set in stone. Now imagine the new hires finding out that the science/technology isn't panning out as published? It doesn't work the way Bruce Booth and co-workers believed. Why did the investors believe that it worked? 

It is in this area, where the people with power run into trouble with the real boss of us all. Science doesn't care about a Venture Capitalists hopes and dreams. This area, science, is where the poor bastards with new PhDs and bachelor degrees have to work. It is where I worked and one day had to start this blog to deal with the insanity. Where a man who lived around 300 BC was once able to accumulate knowledge and put it down for people to learn from for over two millennia, we have arrived at a place where science is taken as a secondary function of a science based endeavor. The analysis of Bruce Booth on why On-Q-ity failed was an "Elements" of its own. It offered up the view from Cargo Cult leadership on what matters. The reasons for jumping in were not scientific. The reasons for failing were. 

Will this simple message ever make its way to the world? Science matters.

One axiom from my own axiomatized deductive structure before I end. Biotechnology in the pharmaceutical industry is plagued by unsubstantiated (non-reproduced) published research. At the core of both the bad science and the bad investment is the desire for money and power. A career, a new car, a busy schedule for speaking engagements... The signs of a true success. They are all very nice but only a true scientist would shun them all if it meant being a bullshitter. Perhaps among the ranks of our janitors and farmers are the scientific sages of our modern world. Our brightest minds are lying dormant until the day comes when pure thought returns to fashion. 

Negative Results On-Q-ity

Bruce Booth has been an interesting blogger on the topic of biotechnology. He is the recovering scientist turned early stage VC. I've made the analogy that he is like the recovering gambler turned gambling advisor. You take the risks, I'll mitigate mine by using your money. Bruce wrote this post about a company he and his co-workers backed with other people's money. It failed and the post mortem was indeed the product of a most clever wordsmith.

Bruce's original thoughts on the investment opportunity:

• “Hot space at nexus of two exciting themes in cancer and personalized diagnostics”
• "Strong team with a track record in building diagnostic businesses"
• “Robust financials with attractive recapitalization” 
• “Very strong intellectual property”
• Perception of “multiple possible exit paths via either diagnostic or oncology-focused pharma M&A or possible IPO”

Sexy, sexy, sexy!!! This company had a hot space, strong team, robust financials, strong IP and an exit door. What would the Reproducibility Initiative have added to this assessment? What Bruce was buying with other peoples money was a service. The money maker was suppose to be a cancer diagnostic service. Where in the above analysis do we hear anything about the science?

We here at the CCS believe that the number one reason most companies fail, and most projects in any company (successful or not) also fail is due to Cargo Cult Science. Whatever the reasons behind practicing CC science, it will not pan out in the end. The cargo planes will not land. In spite of this fact, money can still be made. This is what motivates Bruce to get the hell out of science to become an investor. I don't blame him, I got out too. The only difference is that I wouldn't waste any money on any biotechnology unless the technology is based on sound science. From that point the technology must then work as advertised. From that point you can start to look into the hot spaces where a strong team and IP can... MAKE MONEY!

After all... it is a business. Here is an easier decision, one would hope. The Office of Research Integrity (ORI) ruled that Andrew Aprikyan committed scientific misconduct. He no longer works at UW. Where does he work? He works at Stemgenics Inc. Stemgenics Inc. has recieved $180,000 USD in the form of a small business loan titled, SBIR Phase I: Membrane-Permeable Nano-Materials for Cell Reprogramming Solutions. The likes of Bruce Booth would probably have to pass in Stemgenics Inc. but the NSF likes what they see. In time Andrew Aprikyan could pull the wool over the diligent eyes guarding VC capital, except for the strong management team. Having a founder who has been ruled against in a misconduct case at the ORI should be a red flag for a venture capitalist. You may be able to hide this fact, but is the scientific foundation equally as shaky as the founder?

Ultimately, you can analyze the founders and the company all you want and you can easily fail. A better analysis would be focused on the science. Rather than doing what Atlas Venture did, hire a team at The Reproducibility Initiative to look into the science behind the technology. Hire anyone who has a laboratory and the ability to test the theory. If you think venture capitalists have a scientific method for picking winners, you would be mistaken. Science often takes off on paths that lead nowhere. That is part of the fun. The question for investors is "When?" is a business plan ready, from the scientific perspective? Without reproducibility, you are going to do what Atlas Ventures and On-Q-ity did. You will spend millions and millions more having the new lab staff try and validate the science. Chances are they will fail and your money is gone. 

Positive Results

I'm obsessed with the Amgen Study. It is true that the results are what I expect thus my own bias is satisfied. I don't like the results, but they support my belief in this whole cargo cult idea. So I went to set up the link on a recent post and I spent the time to read some of the comments. The first comment I read mirrors my thoughts.

Publication bias is a problem in all fields of research. The results of a paper should actually receive zero weight in the evaluation of its quality, otherwise there is the motivation to cherry-pick the data that give the most impressive result. The measure of quality should be the way the results were obtained – size of sample, experimental procedure, endpoints used. Ideally the reviewers of a paper should not see its results at all, only the description of the experiment.

But then there are a number of comments highly critical of the paper. What did these people not like?

The claims presented here are pretty outlandish. Particularly relevant to "Hematology and Oncology" we now know that mice housed under different conditions with different microflora can have vastly different outcomes in any model, not just cancer. To suggest academic incompetence or outright unethical behavior is offensive, and is a particularly narrow view of why experiments are difficult to reproduce.

This commenter seems to be defending non-reproducible results. Scrolling down we get a comment on this comment.

Reproducibility is the crucial part of scientific method. If indeed "mice housed under different conditions with different microflora can have vastly different outcomes in any model" (quot. from one of the comments above) then I wonder if mice presents a useful model-study that can be used outside a particular lab and on which further investigations can be built upon outside that one lab. 

Perhaps the commenter critical of the Amgen Study does not have a firm grasp on what makes for good science. The natural urge to simply defend could come from the human condition of favoring positive outcomes. As the first commenter points out, results shouldn't matter. That is a very interesting concept that not everyone will understand.

But all of this leads me to the latest TED Talk from Ben Goldacre. He too brings up the Amgen Study. He talks about this idea of positive results and our tendency, as humans, to favor them even when negative results are the accurate results. Hiding them leads to trouble, scientifically speaking.




Again, Feynman CCS

One example of the principle is this: If you've made up your mind
to test a theory, or you want to explain some idea, you should
always decide to publish it whichever way it comes out. If we only
publish results of a certain kind, we can make the argument look
good. We must publish both kinds of results.
 
I say that's also important in giving certain types of government
advice. Supposing a senator asked you for advice about whether
drilling a hole should be done in his state; and you decide it
would be better in some other state. If you don't publish such a
result, it seems to me you're not giving scientific advice. You're
being used. If your answer happens to come out in the direction the
government or the politicians like, they can use it as an argument
in their favor; if it comes out the other way, they don't publish
it at all. That's not giving scientific advice.