Study: Many courtroom psychology tests are unreliable

We have all seen some so-called psychology tests that aren’t worth the time spent to take them. They appear in magazines or online quizzes. They aren’t intended to diagnose mental health problems. They’re meant for fun, or whatever insight you may find. 

There have been a lot of psychological tests over the decades, and not all of them are valid. For example, you may have heard that lie detector tests are no more likely than chance to detect lies. It’s true. Some tests are so old they have never been validated by practitioners of the science. Others are so new they are still unproven. 

In court, it really matters whether a test is backed up by science. In a criminal case, someone’s freedom or even life could depend on that test. 

In order to ensure that only scientific tests are admitted into evidence, courts have come up with rules for when scientific or technical evidence should be allowed. Generally, the rule is that this evidence should only be allowed when it is based on scientific evidence and is widely accepted within its field of study. 

But how good are courts at differentiating between junk science and real science that could have a material impact on someone’s case? 

Not very, according to a recent study. Researchers at Arizona State University pored over 876 U.S. court cases that occurred between 2016 and 2018 and contained psychological tests as evidence.  

What they found was disturbing. Of all those tests that were admitted as evidence in court cases, fully a third had never even been reviewed in any of the field’s most prominent manuals. Of the two-thirds that had been reviewed, only 40% were graded unfavorably and could not be said to be widely accepted by the field of psychology. Almost a quarter of the tests were deemed altogether unreliable. 

For example, the researchers found that the second most common psychological test being admitted into evidence was the Rorschach “ink blot” test. While this test, which was developed in 1921, has supporters, most scientists say it is too ambiguous and the meaning is subjective. 

The system needs to do a better job of vetting psychological tests 

Judges need to do a better job of keeping out unsound tests. Attorneys need to do a better job of challenging untrustworthy tests and keeping them from being admitted as evidence. And the field of psychology needs to do a much better job of policing itself so that the courts can tell which tests are worthwhile and which should be disregarded.