My 23andMe Results: Using Predictive Models for Genetic Testing

NOTE: Getting Advice About Genetic Testing

There is a big difference between saying that a mutation has a statistically significant association with a trait and saying that a mutation has strong predictive power for a trait.  To illustrate this point, I’ll focus on the 23andMe prediction for eye color.

One of the top 5 traits shown in my trait overview is “eye color.”  My eye color was predicted to be “likely brown” (which was an accurate prediction).  However, I wanted to look at this report more carefully because I remembered Francis Collins mentioning that 23andMe incorrectly predicted his eye color in “The Language of Life".

I don’t know Francis Collins’ genotype, but I did notice a potential problem when I looked at the section for “Your Genetic Data”

Genotypes for rs12913832 (from 23andMe)

	Percent Brown	Percent Green	Percent Blue
AA	85%	14%	1%
AG	56%	37%	7%
GG	1%	27%	72%

My genotype is AG (shown in red above).  I was correctly predicted to have brown eyes, but I actually only have a 56% chance of having brown eyes.  This means that the AG genotype is roughly as accurate as a flip of a coin at predicting individuals with brown eye color.  In my opinion, I don’t think this should have come up as a top prediction, and I think it probably would have been better to flag this SNP as “not predictive” for this particular genotype.

Now, don’t get me wrong – I don’t think people should only have access to information about their highly predictive SNPs.  In fact, I would want to be able to know the frequency of my genotype if it significantly varies for different traits.  However, I don’t think information like my predicted eye color should have been something that caught my eye within 5 minutes of viewing my results.

In general, I think it would be useful to give scores to predictions in the same way that stars are given for disease risk.  Ideally, it would be nice to provide all the relevant information (such as overall accuracy, sensitivity, specificity. positive predictive value, negative predictive value, etc.) about these predictive models in a table, but in practice it might be better to focus on one or two features for simplicity (especially when dealing with traits that are not binary).   In order to distinguish between predictive scores and reproducibility/statistical scores (which I what I would call the 4 star system), perhaps SNPs with a positive predictive value (PPV) greater than 75% get a bronze circle, SNPs with a PPV greater than 85% get a silver circle, SNPs with a PPV greater than 95% get gold circle, and all other SNPS get labeled as "not predictive."

These calculations become trickier when considering diseases that are significantly influenced by multiple SNPs, and this is where building a predictive model (using SVM, CART, etc.) could really be helpful in providing individuals with the most accurate predictions.  In fact, these predictive models need not only consider genetic information and can also be used for non-genetic / environmental risk factors like weight, family history, blood sugar, blood pressure, etc. (which I mentioned in my second post).

Unfortunately, there is no absolute best way to build a predictive model using different SNPs (and/or non-genetic information).  Based upon my experience, I think regression-based models do a good job of providing probabilities that individuals have a particular trait, and very strong associations should have similar results regardless of which machine learning technique is used.  However, I don’t know if a single tool will be appropriate for creating all predictive models, and I’m sure there are some traits for which no good predictive model can be created.

Since there is not a lot of precedence for clinically-relevant predictive models incorporating genetic and non-genetic information, I think 23andMe has a great opportunity to experiment with this for their trait predictions.  Since 23andMe is the largest DTC genetic testing company, they will have the most incoming data that they can use as validation sets.  If they are worried about how easily individuals will interpret these results, perhaps a separate “experimental” section can be providing this type of result (just like I think it might be best to test these models for the "trait" section before incorporating these results into disease risk and drug response results, which people are more likely to use when making medical decisions).  Also, I should acknowledge that I don't know what's going on behind the scenes at 23andMe (for example, I don’t know how 23andMe is currently combining SNPs for disease risk etc.), so this may have be something they have already started to investigate.

Finally, I would like to close this 3-part post by emphasizing that I was generally pleased with my 23andMe results, and I have only provided constructive criticism because I want these results to be as clear and accurate as possible because I think 23andMe has the potential to be an invaluable resource to empower patients to utilize their genetic information to the fullest extent.