Wednesday, December 4, 2013

Who was the Primary Audience for the FDA Warning to 23andMe?

NOTEGetting Advice About Genetic Testing

The other day, I put together an informal survey about public opinion on the FDA warning to 23andMe and the subsequent class action lawsuit against 23andMe.  I posted links to this survey on Twitter, Facebook, the 23andMe Community discussion group (sorry, link only visible for 23andMe users), and in the comments for two articles describing the class action lawsuit (see specific links here and here).

Since it looks like the response rate has plateaued, I am presenting the results for everybody to see in this blog post.  By the way, I don't work for 23andMe, so I can't provide authoritative "right" answers to some of the questions.  However, I will always provide my own answer.

Question #1

Pretty straightforward.  I wanted to see how responses compared for 23andMe customers versus everybody else.  Almost all responses came from 23andMe customers.

I responded "Yes"

Question #2

Most people think 23andMe is at least as good as other companies.  Although the response rate is limited, I think the 2 individuals that thought 23andMe was "less accurate" provide some potentially interesting information that I will describe in more detail throughout the post.

I said "Comparable".  For SNP calling, I am assuming that most companies are using customized Illumina SNP arrays.  If this is the case, the sequencing quality should be practically identical, regardless of where you go.

Question #3

The vast majority of those surveyed don't think they deserve a refund (which is the basis for the class action suit).  If you're counting the answers for each question, you may notice that 2 people that didn't buy a 23andMe kit accidentally answered this question (both answering "No").  So, really only 45 23andMe customers answered "No," but I think the most important point is that only 2/47 customers expressed concern about their results.

The two individuals that said "Yes" are the same two individuals who thought 23andMe was less accurate than other direct-to-consumer genetic testing companies.  Again, I only have a small sample size, but this may indicate that FDA warning has caused a lot of concern about the quality of 23andMe results that didn't previously exist.

If this is true, this is important to keep in mind because I think it represents a misinterpretation of the FDA warning.  I think it is safe to say that the main message was something along the lines of "Hey 23and and Me - I want to learn more information about your tests, please reply to us ASAP".  This is not the same as "Hey current and existing customers - we are aware of serious problems with 23andMe, so you should not buy this bogus test".  My current theory is that confused customers are taking the later interpretation.

Moreover, this appears the be the same misconception implied by a blog post trying to recruit people for the class action lawsuit:

More specifically, the blog post says "23andMe, it turns out, has been under investigation since 2008 by the FDA".  While technically this is probably true, this seems to be imply that 23andMe has been singled out for suspected wrongdoing for 5 years.  However, this is not the case.  The first news from the FDA I recall hearing was in 2010, where a warning was issued to all the major direct-to-consumer (DTC) genetic testing companies (in fact, I believe Pathway Genomics was issued that warning before 23andMe).  The issue is that these companies are providing an unprecedented service that requires thinking about regulation in new ways.  Getting FDA approval is time-consuming in normal circumstances, and the process for 23andMe will take even more effort.  This is the nature of the long-term "investigation".  The FDA warning last month relates to "510(k)s for PGS on July 2, 2012 and September 4, 2012".  We know that 23andMe was not prompt in its responses since that time.  I would consider this a foolish mistake, but 23andMe can still eventually provide a product that is officially deemed acceptable by the FDA.  In fact, I would bet this will eventually be the case.

If you're still wondering, I said "No".

Question #4

My own opinions have changed around a bit on this one.  As you might guess from the question design, I've assume the true cost to provide the test was greater than the amount charged to the customer (and I choose $100-$500).  Namely, I know 23andMe has a lot of strong financial backing, including NIH funding for their research (which I hope can also help allay concerns caused by the FDA warning: clearly, the folks in this other branch of the federal government are cool with 23andMe).

I have become less certain about the costs recently when I found out that AncestryDNA also provides a genetic test for $99 (click here to see the details from my blog post on that).  My assumption was this company has less extra funding than 23andMe.  The 23andMe array provides ~50% more SNPs than the AncestryDNA sample that I looked at, but I'm not sure how much this really makes a difference in cost for the array.  That said, 23andMe probably has a lot more work to do on the interpretation side (AncestryDNA only provides ancestry results) as well as work with surveys to conduct genomics research.  So, I would probably still give the same answer in the end, but I am less confident in my choice.

To a certain extent, these details don't really matter too much: I would consider $99 dollars to be a great deal for the genetic information alone (even without interpretation from the vendor).  It is safe to say 23andMe is not making huge profits from the collection and processing of DNA samples.

Question #6 (Yes, #6 - I'll explain)

One of my concerns is that I've seen a lot of information on the Internet that I found to be inaccurate or misleading (even from generally respected sources of information), and I was concerned this was causing an exaggeration of concern in potential or existing 23andMe customers.

This question is a follow up for a question where I asked what articles / blogs people read reading the FDA warning and/or the lawsuit.  I'm not showing those results here because they are hard to read.  There was basically just a bunch of check boxes next to hyperlinks (for sites containing both positive and negative opinions about 23andMe).  20/51 individuals didn't read any of the same links that I did, and the follow up question showed that most people didn't really care what they read on-line from these types of sources.  Neither of the individuals desiring a refund read any of those specific links.  One said "No" to this question, the other said "Maybe".  Obviously, they learned the news from somewhere (and again, we're only talking about 2 people), but I did feel better that some of the specific sources that concerned me were probably not a huge deal.

I answered "No" for this one.

Additional Results

I originally used all 10 allowed questions for the free survey, but I ended up deleting 4 of those questions.  Basically, I was also interested in learning other stuff:  What to people think the terms "true positive" or "false positive" mean (with respect to 23andMe results)?  How confident are customers in their quality of results (raw sequencing data, interpretation of results, etc.)?  What to people think the error rate is for the sequence data?  (BTW, I would use this comment / post to argue the error rate less than 0.01%)  What is the perceived difference in quality for the health versus ancestry results?  How familiar are 23andMe customers with the scientific concepts being used and the limitations to the results?  However, I ended up creating confusing questions that jumbled these issues together.

I bring this up because I was impressed by how quickly I got a response from the community of 23andMe customers.  Within hours, I had users point out that the questions couldn't be properly answered without more specific information.  In fact, I received strong worded complaints due to this problem in designing the survey.

In other words, the process of running the survey made me feel more confident in the ability of 23andMe customers to be able to handle and critique their results.  Of course, I have read studies showing this to be the case (at least most of the time), but the act of collecting the data itself provides an experience that went beyond reading statistics in a scientific publication.

Concluding Remarks

The bottom line is that I feel insulted a class action lawsuit is being sought with the assumption that all 23andMe customers deserve a refund (I assume that assumption is necessary, based upon the requested $5 million dollars).  I would never ask for this, and I strongly oppose the lawsuit.

Of course, it would be better if my survey included a larger number of customers.  Hopefully, there can be a more authoritative inquiry into this matter.  My gut says that most customers are satisfied with their 23andMe results and those that are concerned about their results have over-estimated the severity of the potential problems, and I think this limited survey supports that theory.  In the very least, I hope it encourages discussion about this topic and eases some customer concerns.

If I get significantly more responses, I will update this post.  However, I'm not expecting the final count to be much higher (and I can't actually collect more than 100 total responses).

Additional Analysis of AncestryDNA Data

NOTEGetting Advice About Genetic Testing

I have a friend that asked me what extra information she could get from her raw AncestryDNA data.  I've studied by own 23andMe data extensively, so I thought it would be useful to show what tools can also be applied to raw data from other genetic testing services.

I have not purchased an AncestryDNA kit for myself, so all the analysis that I performed was for somebody else.  Therefore, I will not mention any specific results from this analysis.

First, you may find it useful to convert your raw data to 23andMe format.  You can use this Perl script ( to convert your file.  The Perl script can be run using instructions similar to those provided here.

Here are some potentially useful tools to learn more from your AncestryDNA data:

1) Interpretome - free, but requires 23andMe format file
2) Custom scripts - free, but requires 23andMe format file (and probably some comfort with programming)
3) Promethease - can directly use raw AncestryDNA file, but it costs $5.  I would probably recommend trying to use the free options first.

At first, I wasn't certain how well these strategies would work.  For example, I thought AncestryDNA might focus on non-functional regions that wouldn't affect disease risk.  However, I think there is a decent amount of additional information that can be gained from the raw data.

For example, all the functions that I tested in Interpretome worked properly.  Additionally, I found 2,581 AncestryDNA SNPs were listed in the GWAS Catalog.  Of those 2,581 SNPs, 1,337 were risk allele the individual that I tested.  This is less than I had for my 23andMe data (3,050 with GWAS Catalog annoations, 1,626 risk alleles within GWAS Catalog variants), but I think it is decent for an assay that don't provide any health reports to the user.

For those that are interested, here is a venn diagram of the overlapping 23andMe variants (V3 23andMe chip, AncestryDNA result from a few months ago):

You can see that the 23andMe chip covers ~50% more of the genome, but the AncestryDNA chip still covers a fair number of nucleotides.  My only real problem is the lack of labeling data from X, Y, and MT chromosomes as such in the raw data.  There are chromosomes listed as "23" and "25," so I initially guessed that "24" is the Y-chromosome.  However, I noticed that randomly selected rsIDs from chromosomes "23" and "25" both came from the X-chromosome.  So, I don't see a simple way to change those into a standard format.

Nevertheless, I think the information from chromosomes 1-22 provide a lot of information to review.  I hope this post helps AncestryDNA customers find something interesting!

Tuesday, December 3, 2013

Survey of Opinions on 23andMe FDA Warning and Lawsuit

I was surprised to see so many negative (and/or confused) responses to what has been going on with 23andMe the past week or so.  So, I would like to try and better quantify what is the true public opinion on the matter.

Click here to take survey

I've never actually done this before...I'll eventually write a post describing the results, but I don't know how long it should take to get a reasonable number of results.  I'm hoping a week, but this will definitely depend on how much people pass along the survey.

UPDATE: You can see the results of this study here:
Creative Commons License
My Biomedical Informatics Blog by Charles Warden is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 United States License.