Clerk evaluates briefs filed in court, grading them on three dimensions: Arguments, Drafting, and Context. The grading reflects factors like how strong the brief’s arguments are, how persuasive the relied upon cases are, and the extent to which the brief cites precedent that supports the desired outcome.

The higher a brief’s grade, the more likely it is that the brief will win. This is true both overall, and across each of the Arguments, Drafting, and Context dimensions. This makes Clerk not only a powerful drafting assistant, but also a valuable predictive tool.

Interestingly, this predictive capability surfaces an important dynamic in our recently published law firm ranking:

1. the best performing law firms were less likely than the worst performing law firms to file briefs that Clerk predicted would lose, and
2. the lower a law firm stood in our ranking, the more likely it was that they had filed a brief that Clerk predicted would lose, and which did, in fact, lose.

A Little Background

In addition to ranking Big Law litigation departments, we recently used Clerk to evaluate pairs of briefs filed in opposition to each other in the California Courts of Appeal. Clerk graded the briefs and then predicted which brief would win. Clerk’s predictions were accurate 72% of the time, which involved correctly predicting 79% of affirmances and 56% of reversals.

While 72% may not be sufficiently accurate for some use cases, that doesn’t mean Clerk’s predictions aren’t handy. To the contrary, they’re incredibly useful. I’ll explain why in a moment, but first, some context.

For several years now, legal prediction has been an exciting and growing field. Most of the research is being done by academics, and the focus is usually on the United States Supreme Court. Much of that work relies on the The Supreme Court Database — a detailed and painstakingly produced database of SCOTUS precedent developed by the Washington University (in St. Louis) School of Law.

Typical approaches to prediction involve training machine learning algorithms based on metadata summarizing the case and its issues. A nice example is the great work done by Daniel Katz, Michael Bommarito, and Josh Blackman. They published a paper last year detailing a machine learning algorithm (a random forest classifier) that predicted outcomes based on metadata such as the reason for granting cert, lower court disposition, issue, and data about the justices such as their individual rates of reversal, rates of dissent, and left-right direction.

The Katz-Bommarito-Blackman model achieved 70.2% accuracy at predicting case outcomes and 71.9% at predicting individual justices’ votes for US Supreme Court decisions. What was especially groundbreaking was their model’s ability to predict outcomes for future cases, a key feature for any real-world use of the technology.

Overperforming and Underperforming

Clerk similarly predicts outcomes in as-yet-undecided-decisions, but in a very different way. The biggest difference is that Clerk relies heavily on the briefs filed in court, which present each side’s arguments detailing why they think they should win. This means that Clerk is not only looking at high-level metadata, but also considering the actual language, arguments and precedent that the briefs are relying upon.

A second big difference is that Clerk doesn’t use a machine learning black box to make it’s predictions. Rather, Clerk’s predictions are based on a highly transparent measure of each brief’s performance.

Clerk grades a brief’s:

• Arguments: measuring the odds of a brief winning or losing based on brief-specific features that cover the strength of the legal arguments, the persuasiveness of the cases relied upon, and the balance of the argumentation.
• Context: measuring the odds of a brief winning or losing based on contextual (non-brief specific) information like the Causes of Action, the Procedural Posture, and the Trial Court Judge.

The mathematical difference between these two grades (Difference = Arguments – Context) quantifies how strongly or weakly the brief is performing, with briefs falling into one of five categories:

• Strongly Overperforming: the brief’s Arguments grade is 10 or more points higher than the brief’s Context grade, indicating that the Arguments score suggests the brief is far more likely to win than the Context score suggests.
• Weakly Overperforming: the brief’s Arguments grade is fewer than 10 points higher than the brief’s Context grade, indicating that the Arguments score suggests the brief is a little more likely to win than the Context score suggests.
• Performing As Expected: the brief’s Arguments grade equals the brief’s Context grade, indicating that the Arguments and Context agree on the odds of the brief winning.
• Weakly Underperforming: the brief’s Arguments grade is fewer than 10 points lower than the brief’s Context grade, indicating that the Arguments score suggests the brief is a little less likely to win than the Context score suggests.
• Strongly Underperforming: the brief’s Arguments grade is 10 or more points lower than the brief’s Context grade, indicating that the Arguments score suggests the brief is far less likely to win than the Context score suggests.

For a given Respondent and Appellant (who, respectively, won and lost at the trial court level), Clerk predicts a reversal if:

• the Respondent’s brief is Strongly Underperforming, or
• the Appellant’s brief is Strongly Overperforming and the Respondent’s brief is Weakly Underperforming

This seemingly simple approach to prediction identifies 56% of reversals. Put differently: software can identify more than half of all reversals beforehand by looking at the quality of the argumentation!

Importantly, this approach is also intuitive. Respondents lose on appeal when they are unable to make as strong a case as would normally be expected of them; appellants win on appeal when they are able to make an unusually strong case for why they should win.

We applied this predictive analysis to the briefs we used in our law firm ranking and found that:

• seven of the twenty firms lost an appeal when they represented the respondent and Strongly Underperformed;
• the lower ranking firms were much more likely to Strongly Underperform when they represented the respondent; and
• the lower ranking firms were much more likely to Strongly Underperform and lose when they represented the respondent.

Playing The Odds

Beyond predicting winners and losers, our analysis also identified several key insights about appellate practice in California.

First, respondents are in the driver’s seat. Not only did they win at trial — which tilts the odds of winning at the appellate court heavily in their favor — their performance is more determinative of the ultimate outcome of the appeal. When a respondent’s brief Strongly Overperforms or Weakly Overperforms, the respondent has a roughly 86% chance of winning on appeal. This is irrespective of what the appellant does. But when a respondent’s brief Strongly Underperforms, their chance of winning drops precipitously to just 33%. The odds are even lower if the appellant’s brief Strongly Overperforms.

Second, while the appellant’s performance is less determinative, it still matters a lot. When an appellant’s brief Weakly Overperforms, the appellant has a roughly 27% chance of winning (irrespective of how the respondent performs). But when an appellant’s brief Strongly Overperforms, their chance of winning jumps to 39%.

So the incentive is there for lawyers to improve their briefs, which is where Clerk’s utility kicks in. Not only does Clerk provide predictive insight into how likely a brief is to win, it also provides guidance as to where a brief’s weaknesses are, and how to improve them.

For example, when Clerk identifies that a brief is relying too heavily on cases that are vulnerable to being distinguished, Clerk suggests relevant cases that are less vulnerable. And when Clerk identifies that a brief is insufficiently forceful in presenting its position (by not citing enough supporting precedent) Clerk provides on point cases that go the desired way.

Moreover, this assistance is equally effective for analyzing an opponent’s brief. Not only can a lawyer evaluate how well their opponent’s brief is performing and how that affects their odds, it’s straightforward to use Clerk to dissect and then attack that brief.

The key for clients and lawyers is to use this technology not just to understand the odds, but also to improve those odds and the resulting outcomes.