quality of evidence?

[electricland]

"Group think" dynamic led to false intelligence on weapons of mass destruction

Intelligence analysts worked from the assumption that Iraq had chemical and biological weapons and was seeking to make more, as well as trying to revive a nuclear weapons program. Instead, investigations after the Iraq invasion have shown that Iraqi dictator Saddam Hussein had no nuclear weapons program and no biological weapons, and only small amounts of chemical weapons have been found.

Analysts ignored or discounted conflicting information because of their assumptions that Iraq had weapons of mass destruction, the report said.

"This 'group think' dynamic led Intelligence Community analysts, collectors and managers to both interpret ambiguous evidence as conclusively indicative of a WMD program as well as ignore or minimize evidence that Iraq did not have active and expanding weapons of mass destruction programs," the report concluded.

Such assumptions also led analysts to inflate snippets of questionable information into broad declarations that Iraq had chemical and biological weapons, the report said.

For example, speculation that the presence of one specialized truck could mean an effort to transfer chemical weapons was puffed up into a conclusion that Iraq was actively making chemical weapons, the report said.

Analysts also concluded that Iraq had a mobile biological weapons program based mainly on the since-discredited claims of one Iraqi defector code-named "Curve Ball," it said. American agents did not have direct access to Curve Ball or his debriefers, but the source's information was expanded into the conclusion that Iraq had an advanced and active biological weapons program, the report said.

Really, these people could stand to learn from the way consensus guidelines on treating medical conditions are developed. They vary a little from group to group, but you'll generally see something like the following (taken from the SOGC's consensus guidelines on using hormone replacement therapy after treatment for breast cancer):

QUALITY OF EVIDENCE ASSESSMENT The quality of evidence reported in these guidelines has been described using the Evaluation of Evidence criteria outlined in the Report of the Canadian Task Force on the Periodic Health Examination. I: Evidence obtained from at least one properly randomized controlled trial. II-1: Evidence from well-designed controlled trials without randomization. II-2: Evidence from well-designed cohort (prospective or retrospective) or case-control studies, preferably from more than one centre or research group. II-3: Evidence obtained from comparisons between times or places with or without the intervention. Dramatic results in uncontrolled experiments (such as the results of treatment with penicillin in the 1940s) could also be included in this category. III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

CLASSIFICATION OF RECOMMENDATIONS Recommendations included in these guidelines have been adapted from the ranking method described in the Classification of Recommendations found in the Canadian Task Force on the Periodic Health Examination. A. There is good evidence to support the recommendation that the condition be specifically considered in a periodic health examination. B. There is fair evidence to support the recommendation that the condition be specifically considered in a periodic health examination. C. There is poor evidence regarding the inclusion or exclusion of the condition in a periodic health examination, but recommendations may be made on other grounds. D. There is fair evidence to support the recommendation that the condition not be considered in a periodic health examination. E. There is good evidence to support the recommendation that the condition be excluded from consideration in a periodic health examination.

Assess the quality of the evidence, and show how strongly you feel about each recommendation. Doesn't seem that complicated, now does it?

Comments

[headgardener.livejournal.com]

Handy methodology for a wide range of fields. Yes, someone should tell 'Intelligence' (as they call themselves) about it

[adrian-turtle.livejournal.com]

What do you think about the general trend of medical thinking moving away from their traditional model? They seem to be moving towards untestable self-perpetuating groupthink (such as produced the political decisions about weapons of mass destruction in Iraq) because it's faster and doesn't require statistical thinking about risk.

People sign up to be in clinical trials for new drugs (which might turn out to work better than the old drugs, or might be worse, or might not do anything at all), and then sue if they end up in the placebo group. I don't have a web reference, but there seems to be a lively discussion in the medical ethics community about whether it's still ok to test new drugs by comparing them to old ones or non-drug treatments. Or whether every new drug must be provided to all patients as soon as there's any hint that it might do anyone any good.

[electricland.livejournal.com]

Unfortunately "information from discredited defector that we really really want to believe" isn't anywhere on the list...

[electricland.livejournal.com]

I disapprove, is what I think of that, if it's going on. Would be interested in references if you have them.

The point of testing new drugs against old ones or against placebo is to see if they work as well, or better, or worse, and if the incidence of side effects is comparable or higher or lower. (Duh.) If interim studies show that one treatment is having a significantly better or worse effect, then I'm all for discontinuing early (as was done in WHI). But honestly, people suing for ending up in the placebo group is just dumb and says to me that they didn't understand the whole concept to start with. (They'd presumably be just as likely to sue if they ended up in the active group and had nasty side effects.)

I mean, the clinical trial process is there for a reason...

This is of course distinct from publication bias and drug companies leaning on researchers to suppress poor results, of which I also disapprove strongly.

[texaslawchick.livejournal.com]

That's why informed consents need to be very carefully written and explained to the patient. Patients that are entering clinical trials because they see a perceived benefit should not be entering trials. Of course, the trial shouldn't hurt the patient, but there should never be any expectation on the patient's part that he or she will be receiving the experimental drug.

[electricland.livejournal.com]

Amen! (and much better put.)

(Also, I love that icon.)