8/18/2010

How Do You Avoid Pseudoscience?

I'm working on lectures for my classes on how to think critically and recognize woo. Joseph Calabrese's Legends, Lore, and Lies is one of the texts I'm using to create the lectures.

His guidelines for evaluating claims are worth repeating here (though I suspect regular readers have little need of being reminded of the skeptic's tool kit:


•"Defer to simpler explanations."
•"Insist on exceptional levels of proof for extraordinary claims."
•"Imagine alternative explanations for phenomena."
•"Remember that fraud is common."
•"Remember that people can delude themselves."
•"When possible, devise tests."

24 comments:

Autism Reality NB said...

This is just more silly name calling. What you call pseudoscience are the views of those who disagree with your opinion. This is the same type of drivel that is found each day at Respectful Insolence.

KWombles said...

No, it's not, Harold. There was no name calling here. Nor was there drivel. I'll happily accept a variety of opinions over at the Directory, but I'll just as happily pull your blog from over there yet again if you can't contain your angry rants.

These were the bullet points from the introduction to a textbook for helping students learn how to think critically and evaluate claims for themselves.

Thank you for providing a perfect example of someone who doesn't do that. You've provided a perfect teaching example!

Autism Reality NB said...

Feel free to pull my blog from your directory at any time.

You use insults in place of reason. Insults like calling opinions that do not reflect your Neurodiversity ideology pseudoscience, woo, quackery etc. That is just stating the facts and pointing out your hypocrisy.

It is unfortunate that your students impressions of autism disorders and autism issues will undoubtedly reflect your personal opinons.

KWombles said...

Harold,

Where in this post are there insults? Where in this comment?

See, the person who came here and hurled insults was you.

Judging something to be pseudoscience isn't an insult. It's an informed opinion based on scientific evidence. If it can't be tested, it isn't falsifiable, it's been debunked, it has no logical or scientific rationale, it just might be pseudoscience.

Truly.

Lyn said...

I didn't see any name calling.
it's always good to be suspicious of everything. Especially pseudoscience.
Killing rinos for virility comes to mind.
Folks should not do that.

lifewithasperger said...

Yeah, I gotta say, it's probably okay to call it pseudoscience if the journal that published the study RETRACTED it, or if the doctor who performed the study LOST HIS LICENSE after the board investigating him found he used SCIENTIFICALLY FRAUDULENT and POSSIBLY CRIMINAL means of arriving at his "conclusions".

Lyn said...

Indeed, lifewithaspergers.

Liz Ditz said...

Wow, Harold sure has an autism-centric view of the world.

Woo
*Homeopathy
*Kangen water
*Chiropractic for anything other than low back pain
*Colored glasses to "cure" reading problems
*Applied Kinesiology
*Psychic abilities
*All paranormal claims

The list goes on and on.

KWombles said...

Liz, I thought it rather fascinating that a post that had nothing to do with autism at all was immediately taken the way he did.

And I'm teaching developmental writing this semester, with a critical thinking component; I'm still trying to figure out what he thinks autism has to do with that. The course schedule is available on both the class blog and my website. No autism-specific mentions on that schedule; just a list of the fallacies I'll teach them.

I mean, if he's willing to provide the examples of those fallacies I need, that's awfully sweet of him.

Liz Ditz said...

Also see Novella's post this morning with the ur-woo story arc:


The pattern is depressingly common – take a disease that is not currently cured by science-based medicine. Claim that doctors “don’t care”, or are ignoring treatments that do not make them money, or there is a conspiracy of silence headed by “Big Pharma.” Then offer a snake-oil alternative based upon anecdotal evidence. Sometimes faith and God are thrown in for good measure.


Another familiar pattern is seen in Morgollon's, multiple chemical sensitivity, and electromagnetic sensitivity. -- there's something invisible wrong with me! Doctors won't listen! Woo to the rescue!

Sirenity said...

Wow, Kim, I am not sure where that angry outburst came from (referring to ARNB).
Hugs for you

Autism Mom Rising said...

Not picking sides here between people cuz I like Kim's and Harold's blogs. I will say that I thought Kim's piece was fine, as it was delinating a definition of skepticism.

For myself, I walk a balance between being open to what is at the cutting edge, (because often a lot of what is established science today was once cutting edge and even called woo woo) and being mindful of that sometimes what seems cutting edge is really junk science.

Sometimes scientists are too narrow in their hypotheses and produce studies that end up giving distorted impressions. yet, their studies end up being heralded as seminal. I think Harold D. has brought up a fine example of this recently on his blog concerning the Autism/MRI study that exluded the severely Autistic. Now conclusions will be drawn about the entire spectrum based on those results. Same thing with the GFCF study that was revealed at IMFAR this year. It actually excluded Autistic kids with digestive issues. Who excludes Autistic kids with GI issues from a GFCF study? After all, it is usually only parents with such kids that are reporting benefits from the diet. Yet that study is heralded as some kind of proof that the diet is bunk. My point here is just sometimes studies accepted by the establishment are not always that great and sometimes one's not accepted are ok. The scientific method is great, but only when it has a decent hypothesis with which to work. I think better hypothesises could be crafted if researches actually spent time talking to families.

Thanks Kim for some great food for thought.

KWombles said...

AMR,

Friends, good ones anyway, don't force friends to choose sides. :-)

The skeptics' toolkit should be used on studies, too. Especially on studies. Just because it was peer-reviewed and published doesn't mean it passes scientific muster. Replication is key.

And if the right questions aren't asked, well, not very helpful.

If the question is, does the GFCF diet help children with autism and accompanying GI issues, hasn't been posed and tested by scientists, it should. My understanding, having read the GFCF studies published to date, was that they were asking the question does the GFCF diet help children on the spectrum based on the opiate theory.

Now that that particular question has been answered several times with the answer that the opiate theory is invalid, the question of whether GFCF diet in individuals with autism and GI issues is helpful in lessening GI symptoms and improving functioning should be asked.

Has it not been asked and answered because the question of does removing gluten and casein from the diet of someone with gluten and casein intolerances or allergies lessen GI distress has already been answered and it been extrapolated out in practice to children on the spectrum? Probably. If you have celiac, you have to get rid of the gluten, and studies do show improvement in symptoms.

:-)

Lisa Jo Rudy said...

Here's the question, tho: I know that a mind that's too open lets in flies, and all that...

but how do you know, for example, which is the simplest explanation... (what's simpler: tiny bits of organic matter called chromosomes, so small you can't see them, organize themselves into patterns that are ever so slightly different from the usual patterns, causing ever so slight changes in brain structure, resulting in neurological differences OR vaccines cause autism?)

whether or not a particular researcher has been the victim of a smear campaign or is on the take... (think of Galileo!)

whether or not an age-old medicinal might be surprisingly effective in an innovative application?

Just saying...

Lisa Rudy
www.autism.about.com

KWombles said...

Yeah, Lisa, that may be easier, but it's not perhaps simpler.

Here's the explanation by Calabrese for deferring to simpler explanations: "If crop circles can be made by ordinary people, why invoke supernatural 'vortices' or aliens to account for them? This is sometimes expressed as 'Occam's razor,' which advised that 'one should not increase beyond what is necessary, the number of entities required to explain anything.'"

What is a more logical explanation, that the underlying architecture of the brain can be dictated by the coding of proteins or that one injection of a killed or weakened virus can cause a global reformatting of that wetware, especially when several large studies haven't picked that up and have in fact shown that autism is not increased in those populations who have had thimerosal or the MMR?

On a personal, individual level, I'm going to respect the person's narrative (unless it's clear that it's a narrative that has shifted over time until it's a completely different tale). Arguing with people on their personal narratives is pointless (although it may be fun, and I suspect what drives some folks).

As to the other speculations, for a researcher on the take, let the facts be determined (fraud is common), and for whether a treatment works, let a good, well-designed double-blinded study show whether it works.

The age or length of time a substance has been in use has no bearing on its efficacy in treating disease.

There are plenty of erroneous beliefs that have stood the test of time; that proves only that people have accepted the tale, not the actual veracity of the claim.

Lisa Jo Rudy said...

I agree that Occam's Razor works just fine in the case of crop circles and certain other paranormal controversies.

But I'm just not sure it's the right tool to use to determine questions like "is there a rise in autism?" "do environmental toxins cause autism?" etc. I just don't think one answer is significantly simpler than the other.

Re the "old isn't necessarily good" issue, I agree that it's possible for something to be used forever and yet be useless -- but the world is full of plants, animals and compounds that have been found to be quite useful indeed in treating all sorts of ailments. who knows what might turn out to be helpful for certain symptoms of (or associated with) autism?

Lisa

KWombles said...

:-) Lisa, I wasn't suggesting we use the simpler explanation to determine whether autism rates were increasing or potential treatments.

I think most of those questions would clearly fall under the last item: "When possible, devise tests."

Obviously, we as individuals cannot go around devising all these tests, so that last one would better read, look to see if scientific studies have been conducted, then look to see if they were good tests, then look to see if there was statistical significance.

spectrum times said...

I support your articles Kim,a lot of antivax & "curebie" crowd are afraid of Autism acceptance in favour of perfect world...if you don't like what is said here kids then move on that is your right as you have the right to change the channel on tv.

Autism News Beat said...

LisaJo, what guidelines would you offer for evaluating claims?

Lisa Jo Rudy said...

ANB - I'm not sure how I feel about "imagine alternative explanations" or "defer to simpler explanations" relative to many of the questions raised in the field of autism.

Plenty of people imagine alternative explanations, and you've read all of them - cell phones, aliens, you name it.

And simple explanations for autism don't necessarily seem to be the explanations backed by science. Complex changes in DNA, differences in brain structure, etc., aren't any "simpler" than "environmental toxins."

Finally, while well-designed studies answer the questions they ask, by definition they can only answer very small, very defined questions like... "In a population of men between the ages of X and Y, all of whom are caucasian, from Los Angeles, and diagnosed with Asperger Syndrome, is eye gaze different from a control group during specific types of social interaction?" Yes, those studies are probably useful as very small pieces of a puzzle - but from a practical point of view, they don't mean much to today's families.

I have to say I think claims can be evaluated differently, depending upon their source, what the claim is, what the outcome of the claim might be, and even - in some cases - personal experience.

If a 12 year old says "I saw bigfoot in my garden last night, and here's his footprint to prove it," I'd be pretty skeptical.

If Paul Offit said "I have concerns about the safety and efficacy of this year's flu vaccine," I'd be likely to listen - and very likely to avoid the vaccine on basis that he probably knows what he's talking about when it comes to vaccines.

If I see with my own eyes that my child is benefiting from a risk-free social skills group, I don't worry an awful lot about proof.

If I am trying a risky, expensive treatment that may be woo, I worry a great deal about proof.

Lisa

KWombles said...

Well, if nothing else, this is an excellent dry run for teaching this material; my lecture for this material already included explanations for each of the guidelines and it's readily apparent they are needed.

I note that the following guidelines are being ignored in order to focus (and misinterpret) the first and third guidelines:

"Insist on exceptional levels of proof for extraordinary claims" and "Remember that people can delude themselves."

Being a critical thinker doesn't mean you take the guidelines literally and assume that for every possible question you should, for example think of the simplest explanation possible. Substitute most probable if that makes it more understandable.

Insist that the people making extraordinary claims (like mercury in vaccines causing autism in spite of the studies exonerating it) offer extraordinary proof.

Alternative explanations doesn't mean any hairbrained idea that comes to mind, either. It means are there other more plausible explanations. In other words, is there a more plausible explanation than the one being provided? Think bigfoot. Think chupacabra. Are there more logical explanations that are more likely?

And perhaps most importantly, remember that we can and do convince ourselves of plenty of nonsensical and inaccurate things day after day. How many people still believe that we only use 10% of our brains? How many people still think that ERs and jails are busier on the night of the full moon?

Lisa, the problem isn't by and large with the studies themselves, but with the misinformation people have regarding how and why studies are conducted and what to do with the information generated from a particular study. Studies, in order to be as accurate as possible, must be as precise as possible. Every variable added adds a layer of information to be analyzed that makes it hard to determine the actual effect of the independent variable.

Studies generally don't look at the large picture because every variable has to be controlled. And no one study was ever meant to be the be all-end all. It is a layperson's lack of understanding regarding the cumulative and self-correcting nature of science that leads to people choosing pseudoscience over science. There are rarely quick, fast, or hard answers in science, but quacks provide all of that and usually for a bargain, to boot.

So, focusing on deferring to simpler explanations (doesn't mean easier, faster, tidier, but does mean more logical, practical and reasonable) and on imagining alternative (while taking that to mean nutty) explanations misses the point entirely of being a critical consumer of information.

Lisa Jo Rudy said...

I do understand your point, Kim; my frustration is that even the terms used to describe the critical thinking process seem to me to be fuzzy.

What is the more likely explanation? Sometimes the answer is clear; sometimes it really isn't. Every now and then, for example, I wonder how in the world we figured out that sex leads to pregnancy? So many things happen between the act of sex and the birth of a child that making the link seems just plain silly (lacking the information needed to understand the biology).

What is an extraordinary claim? If I actually saw my child fall apart and lose skills within hours of an event, I wouldn't think it extraordinary to posit a connection between the event and the impact on my child.

I do understand the scientific process and the difficulty with managing variables. In fact, when testing children relative to autism, I find it incredible that variables can EVER be managed. I mean, how many study managers tell parents "your child may not try any other therapy, start any new class at school, eat any new foods, sniff a new carpet, or otherwise engage in experiences that could confound our findings?"

In short, I don't think it's a simple matter to do good science, or to avoid pseudoscience - especially when one is attempting to keep an open mind.

Shakespeare said There is more in heaven and Earth than is dreamt of in our philosophies, and I think he was right.

Alexander Cheezem said...

Lisa,

Taking a nice look at Hill's criteria of causation (I'm pretty sure the paper's available for free somewhere on the 'net) should help.

Augsnod said...

"Another familiar pattern is seen in Morgollon's, multiple chemical sensitivity, and electromagnetic sensitivity. -- there's something invisible wrong with me! Doctors won't listen! Woo to the rescue!"

I can't speak for Morgollon's or EMS, but with multiple chemical sensitivity there is real physiological suffering and not a whole lot of answers. When allopathic medicine doesn't help, people turn toward alternative medicine. They frequently will desperately cling to bizarre ideas for treatment. I don't myself. I have no intention of wasting my time, money, and hope on predators. As it happens though, there ARE allopathic physicians who treat MCS.