[Original post here: How Trustworthy Are Supplements?]

1: AvalancheGenesis writes:

I think the bigger issue is that the industry as a whole sort of exists as solutions-in-search-of-problems...deficiencies really aren't that common, or even meaningfully health-affecting unless dire. (Fairly-arbitrary worldwide differences in target levels of IUs also remains puzzling.) Discerning customers can benefit from targeted supplementation. But that's not the median supplement purchaser, far from it. The median supplement user is more like...my former coworker who claimed he never got colds because he took 1000% vitC pills every single day, or whatever. At some point, the explanatory process for That's Not How It Works At All is just too long, so...let people believe things. Supplements are surely an easier way to sell hope and agency than most options. At least he picked something water-soluble and cared about proper hydration.

Vitamin C probably doesn’t prevent colds in the general population, though some studies suggest it does prevent colds in athletes, and there’s some medium-quality evidence that it might shorten colds a little once you have them.

The supplements I find more interesting are things like melatonin for sleep, ashwagandha or silexan for anxiety, SAMe for depression, or caffeine + theanine for focus. All of these are useful, supported by studies, and good alternatives to medications that some people don’t tolerate well. I’m using mental health examples because that’s the subject I know about, but there are probably examples in other fields too (probiotics for digestive problems).

Some commenters chimed in to discuss supplements that have anecdotally worked for them (1, 2, 3). And Elizabeth’s story here is also a good example of how I think about this.

2: Gbdub writes:

You seem extremely credulous (uncharitably, “gushing like a fanboy”) about MYASD’s claims. Coming into this with no dog in the fight (I take no supplements and this is the first time I’ve heard of this dude), my inclination is to be more skeptical. His claims may be accurate, but he’s also somebody whose livelihood involves selling a premium product to a niche market of Grey Tribe Redditors. And “scientifically serious little guy with edgy Reddit handle rails against the sloppiness and damn-the-consumer profit seeking of Big Supplement” is great ad copy for that niche market. And indeed, you’re eating it up.

To be clear I’m not saying that their factual claims are bullshit. And they are probably spot on about some of the really sketchy Chinese stuff. Just that we ought to be skeptical of their conclusions (particularly those against the generally honest but allegedly less rigorous other companies). Sure, maybe the rest of the industry is being unduly sloppy in their manufacturing process. Maybe the industry is using one standard for expensive mushroom juice that lets them get away with selling less expensive mushroom juice without technically lying.

But given that the evidence for therapeutic efficacy of most of these supplements, let alone evidence that precise dosing of them matters therapeutically, is often pretty sparse, how do we know this isn’t all just a veneer of scientism for marketing purposes? “My snake oil is much more pure and precise than the snake oil sold by greedy, sloppy Big Snake Oil! Sure, it costs a little more, but do you really want to RISK YOUR HEALTH with snake oil that might contain +/- 25% the advertised amount of snake?”

I’ve been reading his stuff for almost ten years now. I guess at some point in reading someone’s online writings for a long time, you start to know them as a person and trust them. Some people who read my blog say they trust me - hopefully not blindly, just in the sense that they know my virtues and deficiencies and assume I’m not totally lying to them. This is how I feel about MYASD too.

I said at the end that it’s probably not a problem if your supplement has +/- 25% of the active ingredient, so I don’t know where we’re disagreeing here.

Also, I will never tire of reminding people that “snake oil” is closely related to fish oil - now recommended for cardiovascular problems, joint pain, mental health issues, etc - and it probably worked just as well. It got a bad reputation because people started selling fake snake oil (technically: oil from omega-3-less rattlesnakes instead of omega-3-rich water snakes), leading “snake oil” to be associated with scams and fakes. Over a hundred years, people forgot the story and started associating it with things that never worked at all. Which brings us back to our original question of whether supplement labeling is trustworthy - if people could have been sure that their snake oil contained only the snakes listed on the label, there would have been no problem!

3: Stephen Pimental writes:

> “The most concerning category was mushrooms, where about 25% of brands used some mycelium (the “roots” of a mushroom, which have fewer health-promoting chemicals than the above-ground part) instead of or in addition to the mushroom itself.”

Paul Stamets openly uses mycelium in his supplements and argues that this is better for health benefits.

I think this is a minority position. There’s a brief discussion about it in the comments here, and here’s an article about the dispute.

4: Diddly (writes Diddly Links) writes (I’m combining two comments):

I'm not sure where I heard this and my Google-fu is weak, but I've heard that there are literal heavy metals in supplements and this is not regulated in the slightest? This is what I found offhand:

- https://www.pbs.org/newshour/health/analysis-some-natural-supplements-can-be-dangerously-contaminated

- The FDA recalled some supplements due to high levels of lead: https://www.fda.gov/food/dietary-supplement-products-ingredients/fda-advises-consumers-stop-using-certain-life-rising-dietary-supplements

- And some random study found that 5% of supplements were above the daily exposure to arsenic in their sample: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0049676

This seems more troubling to me than if the amount is slightly off relative to what is claimed.

The LabDoor and ConsumerLab analyses I mentioned in this post also checked for heavy metals; most of the products were at undetectable levels, and none were at dangerous ones. Still, this was just one or two dozen, and maybe a product needs some level of reputability to even make it to LabDoor, so let’s look at Diddly’s links.

They make three main claims: first, supplements are contaminated with bacteria. Second, one particular supplement company had its products recalled for dangerous heavy metal contamination. Third, a study showed 5% of supplements contained arsenic above a maximum safe level.

The first article says “In one assessment, researchers found bacteria in all 138 products they investigated.” I have trouble figuring out how to think about this. I take supplements sometimes. Many of my friends take supplements sometimes. According to studies, 86% of Americans take supplements sometimes. If all supplements are contaminated with bacteria, but basically nobody ever has supplement-bacteria-contamination-related health issues (source: I hear about this with approximately the same frequency I hear about food poisoning from heavily-regulated chain restaurants), doesn’t this demonstrate that whatever bacterial-contamination-standard is being used is meaningless? Don’t approximately 100% of objects contain bacteria? Food? Drinks? The human body? I guess I would want something that explains what’s going on here more before I care about this.

The second article seems more serious. It’s from 2019. When I Google various permutations of “supplement dangerous lead level”, all the examples I can find of this happening in the US are still about this one case from a supplement company I’ve never heard of in 2016.

In fact, what is Ton Shen Health / Life Rising, the company implicated in this? I’m having trouble figuring this out - when I look it up, it looks like a small chain of acupuncture clinics in Chicago run by “Herbal Master Zhengang Guo”, which is still operating (!) and still selling supplements (!) But the FDA seized 300,000 bottles - how did this one guy produce produce so many supplements? All we have from the FDA, statement is that the supplements recalled “were mostly sold locally in Chicago area in retails stores [sic] and some were distributed to other states through mail orders”.

My best guess is that this is one traditional Chinese medicine shaman guy in Chicago selling some supplements he made himself, mostly in his own office but also partly syndicated through local stores. Between 2016 and 2019 he screwed up and included way too much lead and poisoned three local children. And this is the example everybody gives to indict the entire supplement industry forever, and to try to scare you off from buying melatonin at Whole Foods.

The third article is a study which the comment describes as showing that 5% of supplements had dangerous levels of arsenic, although it’s actually more comprehensive than that and finds similar problems with mercury, cadmium, etc.

The only clue they give us as to which products were the offenders is that 11% of “Chinese, Ayurvedic, and marine” products had high arsenic levels, compared to only 3% of “North American excluding Chinese/Ayurvedic/marine” products. But if we believe them, that rules out that it’s all stuff like Ton Shen.

But I would put this in context - about 33% of spices have heavy metals above safe levels, according to Consumer Reports:

Roughly one-third of the tested products, 40 in total, had high enough levels of arsenic, lead, and cadmium combined, on average, to pose a health concern for children when regularly consumed in typical serving sizes. Most raised concern for adults, too. For two herbs, thyme and oregano, all the products we tested had levels that CR experts say are concerning. In 31 products, levels of lead were so high that they exceeded the maximum amount anyone should have in a day, according to CR’s experts.

I couldn’t find any exact lead and arsenic levels in their report to compare to those in supplements. But CR also finds that many fruit juices have dangerous levels of lead and arsenic, and here I can find a study with a number - the worst brand of apple juice sold in the US around 2010 had 45 mcg/kg (it looks like that brand has since been recalled and modern apple juices are better). The worst North American supplement in the supplement study had arsenic levels of 24 mcg/day. Doing some calculations, you would have to drink about two glasses of the worst apple juice daily to get as much arsenic as you get from the worst North American supplement. As someone who has drunk more than two glasses of apple juice daily at various points in his life, I am going to judge these comparable.

On the other hand, the worst Chinese/Ayurvedic/marine supplement had 2000 mcg of arsenic in it. That’s 2 mg. For context, a medium dose of the antipsychotic risperidone is 2 mg/day. So these supplements contained as much arsenic as a risperidone pill does of risperidone. How do you end up with that much arsenic in a pill? I think some traditional Chinese medicines might, uh, be arsenic. In fact, the study mentions that one traditional Chinese remedy is cinnabar, ie mercury sulfide. Isn’t this how Qin Shi Huang Di died?

So my guess is that taking reputable North American supplements gives you about the same heavy metal risk as spices or juices, and taking Chinese or Ayurvedic supplements can potentially have a much higher risk if you’re not careful. My guess is that taking Ayurvedic supplements that have been processed and Westernized and are produced by Western companies (eg ashwagandha) is fine, but I can’t prove it.

I’m mentioning this comparison because I think people tend to freak out over supplements, and if anyone hears that a supplement might have lead, they tend demand all supplements be banned, or say that you would be crazy to take them. But if they hear that apple juice or spices have the same amount of lead, then they accept it as “well, it’s hard to regulate everything perfectly, and probably most things by reputable companies are safe, whatever”.

But I notice I’m trying pretty hard to defend supplements here and I might be biased. Someone else should sanity-check my work before you take it too seriously.

5: Neil Thanedar of LabDoor writes (on Twitter):

Hi Scott. I'm the founder of Labdoor. I am a big fan of your writing and believe you have made a major error here based on misleading information.

You claimed "Many people have pointed out that LabDoor is a bad company." but cited only one site, Illuminate Labs, that has had a vendetta against us for years.

We have responded to Illuminate's false claims many times, but they refuse to fix their errors.

At this point, we believe Illuminate Labs has an undisclosed partnership with ConsumerLab to recommend them and attack us.

I appeal to your rationality to consider Illuminate's bias and fairly discuss Labdoor's work.

--

Labdoor's research starting in 2012 led to a lot of these famous investigations you cited. We worked with the NY AG's office during their investigation to tell them that DNA barcoding is a weak method for supplement testing and then we provided them with our more accurate data.

Our research has also been the basis for important public investigations cited in Nature, NYTimes ('14, '21), Last Week Tonight, The Guardian, etc.

--

I see you don't like that we deduct points for overages.

We are very clear with our testing and scoring — "Penalties are twice as heavy if a product has less than what they claim (underage) compared to if they have more (overage)."

The vast majority of our users like this system - but we always also share our testing data transparently for free so you can do your own research too.

We are the only site that is fully transparent and shares our testing data for free.

--

I am happy to talk with you about anything related to Labdoor, the supplement industry, or anything else.

My cell phone is 314-757-6345 and my personal email is neilthanedar@gmail.com.

--

If you know any way to help us fight Illuminate's lies, please let me know. These errors are killing us and we've done nothing wrong.

Regards,

Neil

I am grateful to Neil for the extra information, and I feel bad saying anything against LabDoor because I super respect their work. They’re doing something that should be done - testing and rating supplements. I hate saying anything against anybody who’s trying to do a hard thing to improve the world. I personally use LabDoor sometimes and find it very helpful.

That having been said, I don’t think it’s true that it’s “only one site, Illuminate Labs”, who have expressed concern. The Beginner’s Guide To Nootropics, maintained by the r/nootropics moderators and AFAIK unaffiliated with Illuminate in any way, says:

LabDoor supplement ratings should not be considered reliable. They don't post their actual testing results and they have been found promoting products which were completely inauthentic.

MYASD of Nootropics Depot writes:

Labdoor is an affiliate marketing site masquerading as an independent testing site. They use flawed methodologies to give misleading data and profit off people clicking on their affiliate links to buy products. They had a completely FAKE Panax ginseng as their highest value rating product for OVER A YEAR after I showed them data proving it was fake. It was American ginseng cut with corn starch, and I provided them the data on it when they put up their ginseng ratings. It took them a whole year to even mention there was an issue, then they quietly just put a small note on the page saying the results were under investigation. They made affiliate commissions that whole time as everyone bought the fake Panax ginseng by clicking on the links on their rating page... It's not just like it was an innocent mistake. I called them out publicly over it, and we discussed it privately vi PM here on Reddit. They could have removed that product page when I alerted them, but it stayed up. Just insane to me.

Consumerlab is a lot better in my experience. They seem to actually be trying to find the truth on products. I disagree with some of their metrics/methodologies, but those are minor in comparison.

I’m also concerned by reports like this, which I won’t repost because currently they’re only unconfirmed rumors - but I would like to hear LabDoor’s response.

Regarding the overages, I agree that it’s better to penalize something more for an underage than an overage, but I am still not sure this is good communication. Consider for example their bacopa rankings. Just looking at this, I would come away with the conclusion “no bacopa is entirely trustworthy, but at least ND and PE are better than Swanson”. But if I’m understanding right, the whole ranking is entirely driven by overages, and nobody (including the brand getting a C-) has any other problem. I suppose it’s fair to, if everything is exactly the same, rank based on a minor issue, but I think if I didn’t understand what was going on here I would be much more concerned about Swanson than justified.

I admit that some of these complaints are letting the perfect be the enemy of the good. Again, I use LabDoor a lot, I more or less trust their numerical results, and I think they’re a good source for people who understand what’s going on and want the specific results of specific tests. But I also worry that people who just use their letter grades without understanding what’s going on will end up confused.

And for what it’s worth, Calloway Cook comments:

Hello. I'm a bit late on this but I'm the founder of Illuminate Labs. My name is Calloway Cook. I just received word of this article today. I appreciate you referencing our reviews of Labdoor and ConsumerLab.

I want to clarify to readers that we do not have any commercial partnership with ConsumerLab, and that the founder of Labdoor's comments asserting such are baseless and are libel. We have never even spoken with that company; we just believe they have a better service than Labdoor.

For what it’s worth, LabDoor is free and ConsumerLab costs money. I give LabDoor a lot of respect for staying free and realize that this is a principled decision and makes life harder for them. But I keep hearing people I trust saying to be wary of them, so I am wary.

6: Marathon writes:

> "A lot of herbal supplements are similar. If this is your strategy, a 25% labeling error isn’t going to matter much, is it? If my patients get their Lexapro from a sketchy company that actually has only 4 mg in a 5 mg pill, they’re still going to go up to 20 mg or down to 2 mg or whatever it is the end up needing, based on how it affects them."

Wait but I read MYASD's complaints as largely being about poor quality control. Thus the concern isn't just the mean being off, but also that the variance is high in measured products.

High variance is what will mess with the learn-by-doing approach you mention. To be clear, stochastic gradient descent/Kiefer-Wolfowitz algorithms still work! But variance is the enemy, all else equal you want that to be low.

In the 25% off example from the quote, the concern would be if one pill from the bottle is X mg, but another is X*1.25 mg (or it could happen at the batch level, depending on where the QC is poor; this might look like "found the right dose but uh oh, now things are weird again in a way that looks like too low/high of a dosage, and all that really happened was starting a new bottle. But bad enough QC could be pill-to-pill I suppose)

This probably suggests a strategy like you mention, plus being extra cautious around finishing one bottle and starting the next. Depending on the severity of the consequences of a change (like, did it take forever to find the right dose?), maybe even keeping a couple pills from a bottle that worked "to the side" so if really desperate could test them. But honestly I suspect that what is natural is just restarting the 'search' process if things seem weird with a new bottle, which maybe is what happens anyway. So maybe once again human locally optimizing intuition us already doing basically the right thing.

Related, does anyone try to apply stochastic optimization techniques to finding good dosages amounts in the way you describe? May not add much, but maybe there are practical variance reduction insights that would be useful, like figuring out how to "subtract off a stochastic constant" (can't remember the right term). Will chew on this a little.

This is a fair point, thanks.