Simplified flow diagram reflecting the process for evaluating examination results. The diagram illustrates the distinctions between results labeled as “appropriate” vs. “inappropriate,” “justifiable” vs. “not justifiable,” and “correct” vs. “incorrect.”
Credit: Swofford et al. 2024
By Christina Reed
There is one answer forensic scientists sometimes provide after analyzing evidence that confounds lawyers and judges: inconclusive. When referring to evidence that requires comparing patterns, such as with firearm examinations, an inconclusive result means that the analyst was not able to offer a more definitive opinion regarding whether the patterns were made by the same source, such as two bullets being fired from the same gun. The resulting inconclusive opinion can be a legitimate answer, but one that courts around the country have struggled to deal with because it raises even more questions about how to interpret the opinion. When are inconclusive results appropriate? How do inconclusive results impact assessments of error rates? Do inconclusive results carry inculpatory or exculpatory implications? These questions have become a point of debate and disagreement among forensic, legal, and academic commentators.
In an article published in Forensic Science International: Synergy, several NIST authors reviewed the literature relating to these different viewpoints. Ultimately, they suggested that an inconclusive opinion, or any opinion for that matter, is appropriate so long as the analyst properly followed the specifications of an approved method. Further, the authors agreed with and expanded on perspectives that error rates, despite their historical use, are not a suitable metric for representing the validity and reliability of analytical methods that can result in more than two outcomes. This is especially the case with firearm examinations and other forensic science disciplines where an inconclusive opinion may be the appropriate option given the information available to the scientist. In these situations, the use of error rates alone provides an incomplete and potentially misleading representation of reliability for courts to consider. As a way forward, the NIST authors recommended more complete summaries of the empirical validation data relating to the methods used, including information about the extent to which the analysts adhered to those methods in a particular case.
Recently, the Texas Forensic Science Commission (TFSC) reviewed the NIST authors’ recommendations and established a collaborative working group focused on understanding and implementing them in Texas labs. “The question the court is interested in, is whether the firearm actually shot the projectile or not,” explained Lynn Garcia, General Counsel for the TFSC. “Lawyers and judges also want to understand how often forensic scientists are able to accurately answer the question of whether a firearm shot a projectile, so they have a sense of how reliable the analysis is. They want to understand what types of evidence pose particular analytical challenges so they can evaluate its strength (or lack thereof) appropriately. We often see a bit of a disconnect between what forensic scientists are trying to communicate about their analysis, including its limitations and the quality of evidence in the case, and what the lawyers and judges understand,” she added.
As a solution, the article, Inconclusive Decisions and Error Rates in Forensic Science, “digs in much more closely to the actual data — closer to the method that’s being performed: in the case before the state courts where most criminal cases are tried, and in the very labs that serve our court system,” Garcia said. The Commission, she added, found the article “very helpful at a practical level for the end-user — the lawyers, judges and juries — and the practitioners themselves.” One of those practitioners, Garcia confided, told her after he read the article, that the “initial implementation of the recommendations will be difficult, but meaningful change is always difficult.”
The article, authored by forensic science research experts at NIST — Henry Swofford, Steven Lund, Hariharan K. Iyer, John Butler, Johannes A. Soons, Robert M. Thompson, Vincent Desiderio, JP Jones, and Robert Ramotowski — puts empirical validation data at the center of the conversation for how to interpret results. The authors of the article recommend forensic analysts include alongside their results information about their adherence to an approved method and the performance of that method when tested using samples that are most reflective of the quality or conditions of the evidence in the case at hand. Such information helps consumers of those results (e.g., lawyers, judges, jurors) better answer the questions that they are faced with when deciding how to weigh the significance of the result. Specifically:
- What method did the analyst apply when conducting the forensic examination?
- How effective is that method at discriminating between the propositions of interest?
- How relevant is the data describing the discriminability (i.e., diagnostic capacity) of that method (generally) to the examination in the case at hand (specifically)?
These questions directly address how one can judge the reliability of the analyst’s opinion, regardless of whether it is inconclusive or a more definitive assertion. The answers to these questions put into perspective whether the examination method was suitable for the evidence under review (the method’s performance itself) and if the analyst followed that method properly (the analyst’s conformance to that method). While seemingly straightforward, many forensic service providers only report their opinion following an examination of the evidence and do not include information about the extent to which the method they used performed when tested using samples most relevant and representative in terms of quality and conditions to the evidence in the case at hand.
By emphasizing the inclusion of this new information alongside analysts’ opinions, NIST is helping those receiving the information to “effectively interpret the weight of that result from a measurement sciences perspective,” said lead author Henry Swofford. Before this recommendation, most efforts at evaluating limitations relied on error rates. “Error rates are only suitable if all cases seem equally challenging and if experts must answer yes or no,” said Steve Lund. “Otherwise, error rates simply omit too much information relevant to interpreting expert opinions,” Lund added. By moving away from error rates and focusing on more complete validation summaries for the particular method used in the case, factfinders can better asses what weight to give an expert’s inconclusive opinion, or any response for that matter.
“It is the most practical and digestible solution to reporting forensic science performance and conformance data that we have seen,” said Garcia. “Method conformance is also of fundamental interest because it may provide important information about the efficacy of the training and quality assurance systems in the laboratory, and indeed may even serve as an effective tool to highlight specific areas for improvement. Decision-makers in the criminal justice system need this information in an understandable format to do their jobs effectively. We are really excited to work with Texas labs and NIST to implement it and we think they are excited too,” she said.
Further Reading:
Swofford H, Lund S, Iyer H, Butler J, Soons J, Thompson R, Desiderio V, Jones JP, & Ramotowski R (2024) Inconclusive Decisions and Error Rates in Forensic Science. Forensic Science International: Synergy (8):100472. https://doi.org/10.1016/j.fsisyn.2024.100472
Texas Forensic Science Commission (2024) “Justice Through Science.” Final Report On Complaint No 21.27; National Innocence Project, University of Colorado Law School (Houston Police Dept. Crime Lab; Firearms/Tool Marks). Available at https://www.txcourts.gov/media/1459122/complaint-2127-final-report-09042024.pdf (Accessed Jan. 10, 2025).
