Science is a very humbling profession. The fundamental stance of real science is doubting oneself, being skeptic about what one thinks, and questioning everything until you have found enough evidence to move past those doubts (on to some other ones, of course).
One of the ways that junk science differs from real science is the incredible overbearing confidence that junk science has –never wrong, always 100% accurate, never needs calibration, doesn’t need replication and certainly is not to be questioned. All of these attitudes kill off real science, because real science feeds on the skepticism about a result which motivates re-testing, which motivates more data collection, and then re-testing, then tweaking the procedure.
In a 2001 article (Empirical Evaluations which you can download here), I reported… a 100% result for an experiment, and then immediately wrote that this would not hold up over re-testing. Several years later, a forensic linguistics expert was testifying that her method was highly accurate and she mentioned that result of 100%. The cross-examining attorney immediately asked her is she was using the same method (smart attorney) and the expert implied that she was. When I was contacted by the attorney’s office to verify the expert’s claim, I had never heard of the expert. Given that my method relied on using specific software, I knew the expert could not have been using my method; if she had contacted me to use the software, I would have made it available to her, but she had never contacted me. When I read her report, I realized that she was really using forensic stylistics, a collection of techniques also tested in the same 2001 article, with the combined forensic stylistics techniques getting about a 54% accuracy rate. I immediately wrote an affidavit clarifying the situation: (1) that my method was not 100% accurate, (2) my method was not being used by the expert, and (3) the method that the expert was actually using was nowhere near 100% accurate.
Science and law can be good partners for truth-finding when scientists stick to the reality of their results and lawyers understand that scientists use methods that are rarely, if ever, 100% accurate over the long haul of scientific testing and re-testing, and re-testing again.
