Every law student is told repeatedly to check that the cases they are relying on are still “good” law. They may even be told that not using a citator such as Shepard’s, KeyCite, or BCite could be malpractice and multiple ethics cases would support that claim. But how reliable are the results returned by these systems?

Paul Hellyer has published the surprising results of an important study investigating this question. Hellyer looked at 357 citing relationships that one or more of these three citators labeled as negative. “Out of these, all three citators agree that there was negative treatment only 53 times. This means that in 85% of these citing relationships, the three citators do not agree on whether there was negative treatment.” (P. 464.) Some of the differentiation between systems could be attributed to one system incorrectly marking a relationship as negative when it is not. This might be considered a less egregious mistake if one presumes that the researcher would review the flagged case and find no negative treatment, although it is a costly mistake in a field where time matters. However, Hellyer accounts for the false positive (or negative, in this case) problem and the results of his study are distressing.

We are told that the citators are reliable. I, along with numerous law professors and judges, have told students and attorneys that failure to use a citator could lead to anything from a judicial tongue lashing to disciplinary action to malpractice charges. As Hellyer points out (P. 450), the marketing for citators assures us that the systems produce reliable results. For example, KeyCite is marketed as “the industry’s most complete and accurate citator” and that you can be “confident you’re relying on valid law.” Similarly, the Shepard’s product page proclaims, “Is it good law? Shepardize® and be sure.” Bloomberg BNA is less boastful in its promotion of BCite stating, “Easy to use indicators…allow you to immediately (emphasis added) see how other cases have treated your case.”

Let’s look at some more data from Hellyer’s study, which he believes is “the largest statistical comparison study of citator performance for case validation” and the first to include BCite. (P. 450.) In addition to just looking at how the citators labeled the relationships, Hellyer assesses the case opinions to determine the nature of the citing relationship and whether it was correctly labeled by the citator. He differentiates between negative relationships that were not identified in any way and those that misidentified the relationship. An example of this would be if a case was in fact overturned but the citator labeled it as something else, such as “distinguished by.” When Hellyer examined whether the citators agreed on the subset of negative treatment, all three systems agreed on about only 11% of references.

Hellyer’s article is an important read for anyone who relies on a citator for case validation or, determining whether a case is still “good” law. The results are fascinating and his methodology is thorough and detailed. Before delving into his findings, Hellyer reviews previous studies and explains his process in detail. His dataset is available upon request. The article has additional value because Hellyer shared his results with the three vendors prior to publication and describes and responds to some of their criticisms in his article, allowing the reader to make their own assessment of the critique.

Even more interesting than the broader statistics, are Hellyer’s details of specific errors. He acknowledges that omission errors, as opposed to misidentification errors, were unpublished cases that might present less of a problem for attorneys. However, Hellyer goes on to examine the misidentification errors and concludes that all three citators exhibit the greatest issues not in identifying the cases but in the editorial analysis of what the citing relationship means. For example, in Hellyer’s dataset there were four cases later overruled by the United States Supreme Court. All three citators misidentified at least one citing relationship and one of them misidentified three of the four cases as something other than being overruled. Hellyer’s examination of these cases revealed how these misidentification errors can filter through to other citing relationships and create further errors. (Pp. 467-471.)

Analysis of citing relationships, and whether those relationships are positive and negative, is essential to the legal system and reliance on “good” law, or case validation, is the critical first step. Hellyer states that the results of his analysis mean “that when you citate a case that has negative treatment, the results you get depend mainly on which citator you happen to be using.” (P. 465.) This is a stunning assessment of a vital resource that is so widely and heavily relied upon by the legal community.

Editor’s Note: This article is published with the author’s permission, with first publication in Lex JOTWELL.