Study: Major FBI pattern matching method is unscientific

Many common forensic evidence-gathering techniques are less reliable than you may have been led to believe by “CSI.” In fact, some of them appear to have little scientific basis.

In 2009, the National Academy of Sciences issued a report on the state of forensic science in U.S. courts. It found that “[w]ith the exception of nuclear DNA analysis … no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.”

With that in mind, over the past year the nonprofit newsroom ProPublica has been reporting on the reliability, or lack thereof, of various forensic techniques commonly used in criminal cases. Now, a leading forensic image analyst and a postdoctoral researcher in image science have decided to do new, groundbreaking research on each of many forensic techniques to prove or disprove their reliability.

Their work was published recently in the journal Proceedings of the National Academy of Sciences and was also covered by ProPublica.

They started their research with a technique called blue jean wear pattern analysis. This technique is commonly used to match the blue jeans worn by a perpetrator in a surveillance video with jeans owned by a suspect. The idea is that the wear pattern on the vertical seams of every pair of jeans is as unique as a barcode. These “barcodes” can be matched up to positively identify a pair of jeans as the one worn in the video.

Are blue jean wear patterns unique? Can they be positively matched up?

To find out, the researchers took high-resolution photographs of the vertical seams of over 200 pairs of jeans, some new and some used. They then tried to use those photographs to match up with the actual pair of jeans or with other photographs of the same jeans.

To prove or disprove the “barcode” theory, the team documented the wear marks as the FBI would and then scanned the photographs into a computer. This computerization digitally converted the patterns into numeric values so that the computer could calculate how similar or different the jeans in the images actually were.

What they found was surprising. Far from finding it an easy matching job, the computer struggled to differentiate different pairs of jeans from one another. It also struggled to match different photographs of the same pair of jeans. All in all, the computer was accurate less than 30% of the time.

They concluded that “identification based on denim jeans should be used with extreme caution, if at all.”