Hyperion
09-25-06, 10:04 PM
In the News forum, someone posted an MSN story about a study published here:
http://www.ehponline.org/docs/2006/9478/abstract.html
The point of this post is not so much to discuss the study itself - it is hideously flawed, as I will demonstrate below - but to look at a very basic issue that sometimes pops up when looking at scientific studies, or with these sorts of debates in general. Specifically, I am concerned about times when people with a particular viewpoint choose to hype said viewpoint by completely misrepresenting data so as to make it appear as if their view is valid despite the fact that evidence shows the exact opposite.
In the study, the authors attempt to show that there is a correlation between higher blood-lead levels and ADHD. They did this by taking a large number of children, measuring the levels of lead in their blood, and comparing this with reports from parents of both diagnosis of ADHD and medication use (a child needed to have both to be considered ADHD for the purposes of the study).
The researchers claim that their data shows a correlation between children with higher lead levels and ADHD. When I first heard this, I thought this seemed odd, because ADHD does not resemble lead poisoning or any type of heavy-metal induced neurotoxicity. Metal-induced neurotoxicity should present broad symptoms of mental retardation and irreversible brain damage. It would not create symptoms which can progress and regress through a period of developmental changes and which encompass discrete specific functions which are unrelated to IQ. Metal toxicity would also not be treatable with medication, as medication cannot resurrect dead neurons.
So I looked through the study (which is available for free), and I looked at their statistical analysis as well as their clinical analysis. For starters, I noticed that they were looking at lead levels that were half of the generally accepted limit. This should have set off warning bells immediately: even though lead may be a neurotoxin, it is hard to imagine how it could have these kind of effects at levels well below that which is considered dangerous. It implies that they could not find a large number of obvious cases of lead poisoning with high levels who also had ADHD. They claim that the safe limit for lead is 10ug/dl, and that thei children in the study scored between unmeasurably small and 5ug/dl.
Their first major fallacy involves an improper comparison. They looked at children whose lead levels were in the lowest quintile, the bottom fifth of lead levels, between unmeasurable and 0.7ug/dl. They then compared them to the children in the highest quintile, and found that there were more ADHD children in the highest quintile than in the lowest. However, the highest quintile included scores from 2ug/dl-5ug/dl. Lumping such a large difference together tends to obscure evidence, but this wasn't the true fallacy here...the actual fallacy is that they compared children with ADHD to other children with ADHD without actually comparing them to neurotypical children. After all, the real question is where the difference between ADHD and non-ADHD children lies. Also, they don't mention the middle 3 quintiles. Now, these were not accidental oversights, they clearly knew what they were doing...because when they finally mention it at the end, it is clear that an examination of the real issues would make this clear, and that it would be apparent even to our board members who are not professional scientists, physicians, or researchers. I refer specifically to this paragraph, in the authors' own words:
estimates indicate that 21.1% (95%CI: 4.6%, 25.9%) of ADHD cases among children age 4 to 15
were attributable to having a blood lead > 2.0 g/dL. This corresponds to 290,000 excess cases
of ADHD among U.S. children age 4 to 15 (Table 3).Here we finally have a case where we can compare lead levels in people with ADHD to those of the general population. Now, levels above 2ug/dl are the top quintile, as I mentioned earlier. This means that around 20% of the total population of the study had lead levels in this range.
And 21.1% of kids with ADHD had lead levels in the top quintile! The percentage of kids with ADHD who had lead levels were almost identical to those of the general population. This is not only the direct opposite of what the authors implied in the beginning, not only does it fail to imply lead as a cause of ADHD, but it actually seems to show that ADHD is completely unrelated to blood-lead levels, because the number of ADHD persons with top-quintile blood lead levels was exactly what one would expect to see in the general population if lead were not a factor.
And this is one way in which someone can lie with statistics. Even non-physician friends and family members have laughed out loud when I quoted that finding to them...because it is just that blatantly obvious that these researchers were either trying to claim something that they knew wasn't true, or else they were really #%$ing incompetent...1000 monkeys at 1000 typewriters would take millenia to write something that inane and pointless.
This is why it is always important to understand where the basic question lies, and whether the questions being asked are relevant to the important point. The question of the differences of lead levels solely between people with ADHD is irrelevant, it is apples to oranges. The important question was whether there was a greater percentage of people with ADHD at the higher lead levels than in the general population.
That the researchers chose to ignore the second question, including ignoring their own data that showed that lead was unconnected to ADHD is inexcusable. I doubt that they were ignorant of what the evidence showed...rather I wonder whether they may have been attempting to justify pre-existing beliefs that neurodevelopmental disorders are caused by pollutants. Possibly they were hoping for money from a lawsuit, or maybe they were attempting to appease parents who are desperate to believe that something external must cause neurodevelopmental disorders because it gives them something either to blame or it offers them the false hope that there is something that they can do to prevent the disorder.
Whatever reason, this is a clear example of lying with statistics, and this is why so many people insist on looking at evidence rather than simply believing anyone at face value.
http://www.ehponline.org/docs/2006/9478/abstract.html
The point of this post is not so much to discuss the study itself - it is hideously flawed, as I will demonstrate below - but to look at a very basic issue that sometimes pops up when looking at scientific studies, or with these sorts of debates in general. Specifically, I am concerned about times when people with a particular viewpoint choose to hype said viewpoint by completely misrepresenting data so as to make it appear as if their view is valid despite the fact that evidence shows the exact opposite.
In the study, the authors attempt to show that there is a correlation between higher blood-lead levels and ADHD. They did this by taking a large number of children, measuring the levels of lead in their blood, and comparing this with reports from parents of both diagnosis of ADHD and medication use (a child needed to have both to be considered ADHD for the purposes of the study).
The researchers claim that their data shows a correlation between children with higher lead levels and ADHD. When I first heard this, I thought this seemed odd, because ADHD does not resemble lead poisoning or any type of heavy-metal induced neurotoxicity. Metal-induced neurotoxicity should present broad symptoms of mental retardation and irreversible brain damage. It would not create symptoms which can progress and regress through a period of developmental changes and which encompass discrete specific functions which are unrelated to IQ. Metal toxicity would also not be treatable with medication, as medication cannot resurrect dead neurons.
So I looked through the study (which is available for free), and I looked at their statistical analysis as well as their clinical analysis. For starters, I noticed that they were looking at lead levels that were half of the generally accepted limit. This should have set off warning bells immediately: even though lead may be a neurotoxin, it is hard to imagine how it could have these kind of effects at levels well below that which is considered dangerous. It implies that they could not find a large number of obvious cases of lead poisoning with high levels who also had ADHD. They claim that the safe limit for lead is 10ug/dl, and that thei children in the study scored between unmeasurably small and 5ug/dl.
Their first major fallacy involves an improper comparison. They looked at children whose lead levels were in the lowest quintile, the bottom fifth of lead levels, between unmeasurable and 0.7ug/dl. They then compared them to the children in the highest quintile, and found that there were more ADHD children in the highest quintile than in the lowest. However, the highest quintile included scores from 2ug/dl-5ug/dl. Lumping such a large difference together tends to obscure evidence, but this wasn't the true fallacy here...the actual fallacy is that they compared children with ADHD to other children with ADHD without actually comparing them to neurotypical children. After all, the real question is where the difference between ADHD and non-ADHD children lies. Also, they don't mention the middle 3 quintiles. Now, these were not accidental oversights, they clearly knew what they were doing...because when they finally mention it at the end, it is clear that an examination of the real issues would make this clear, and that it would be apparent even to our board members who are not professional scientists, physicians, or researchers. I refer specifically to this paragraph, in the authors' own words:
estimates indicate that 21.1% (95%CI: 4.6%, 25.9%) of ADHD cases among children age 4 to 15
were attributable to having a blood lead > 2.0 g/dL. This corresponds to 290,000 excess cases
of ADHD among U.S. children age 4 to 15 (Table 3).Here we finally have a case where we can compare lead levels in people with ADHD to those of the general population. Now, levels above 2ug/dl are the top quintile, as I mentioned earlier. This means that around 20% of the total population of the study had lead levels in this range.
And 21.1% of kids with ADHD had lead levels in the top quintile! The percentage of kids with ADHD who had lead levels were almost identical to those of the general population. This is not only the direct opposite of what the authors implied in the beginning, not only does it fail to imply lead as a cause of ADHD, but it actually seems to show that ADHD is completely unrelated to blood-lead levels, because the number of ADHD persons with top-quintile blood lead levels was exactly what one would expect to see in the general population if lead were not a factor.
And this is one way in which someone can lie with statistics. Even non-physician friends and family members have laughed out loud when I quoted that finding to them...because it is just that blatantly obvious that these researchers were either trying to claim something that they knew wasn't true, or else they were really #%$ing incompetent...1000 monkeys at 1000 typewriters would take millenia to write something that inane and pointless.
This is why it is always important to understand where the basic question lies, and whether the questions being asked are relevant to the important point. The question of the differences of lead levels solely between people with ADHD is irrelevant, it is apples to oranges. The important question was whether there was a greater percentage of people with ADHD at the higher lead levels than in the general population.
That the researchers chose to ignore the second question, including ignoring their own data that showed that lead was unconnected to ADHD is inexcusable. I doubt that they were ignorant of what the evidence showed...rather I wonder whether they may have been attempting to justify pre-existing beliefs that neurodevelopmental disorders are caused by pollutants. Possibly they were hoping for money from a lawsuit, or maybe they were attempting to appease parents who are desperate to believe that something external must cause neurodevelopmental disorders because it gives them something either to blame or it offers them the false hope that there is something that they can do to prevent the disorder.
Whatever reason, this is a clear example of lying with statistics, and this is why so many people insist on looking at evidence rather than simply believing anyone at face value.