The Roman god Janus looked both ways at once. That’s an apt metaphor for the divergence in how lawyers look at the use of data in management decisions. Some lawyers look askance at data being used to augment decisions; others look with favor on it. The more clearly that lawyers understand the conflicting bases for their own views and those of their colleagues, the more adroitly they will deal with data in decisions.
First we’ll consider arguments of opponents—or, to be less extreme, lawyers who are reluctant to overdo data in decisions. Then we’ll hear the arguments of proponents—those interested in using data more.
Doubters marshal a variety of arguments. Consider a handful.
Contra Metrics
Leadership is an art more than a science. Leaders and managers make decisions that draw on experience, creativity, judgment and courage in ways that overpower dry statistics. Sure, some numbers filter into the mix of factors weighed, but good decisions come from the qualitative gut, not the quantitative graph. This view showcases thinking as holistic synthesis more than analysis. Those who doubt the efficacy of analytic decision-making point out that the real world is not digital. Decisions take into account many intangibles, a sense of what the future holds and a conviction that we know certain things as true. This position finds support in decision theory and epistemology.
Those who resist the encroachment of predictive analytics emphasize the cost of gathering figures, especially attorney time, and the dollars consumed by cleaning the data and fashioning tables or charts. The expense of acquiring data for a decision can be calculated, but the benefits are elusive and projected. This argument is founded in economics.
Skeptics also question predictive analytics because numbers are always gamed. Once you start collecting metrics, people figure out how to look the best in light of them. Skeptics might reverse the oft-quoted saying, “You can’t manage what you can’t measure,” because once you start making decisions based on measurements, you will start manipulating the numbers toward the result you desire. If dispute resolution times are reported and evaluated, you can bet a speed-up will happen, but under the hood, who knows what finagling took place? Goodhart's Law holds: “When a measure becomes a target, it ceases to be a good measure.” The foundation of this argument is psychology.
Data doubters argue that lawyers work in a world of hierarchy, and if you let the nouveau riche number jockeys storm in, well, après moi le déluge (des statistics). It will upset the established hierarchies. Who can make the final decisions if tables of numbers and graphs are always being thrown up as influential? We need the experienced gray-hairs who have earned their privilege to make the tough judgment calls. Facts and figures in PowerPoint slides undermine rank and power. Data favors democracy, and the aristocrats of law fear it. This argument draws its strength from a political view of organizational power.
Those who disparage data sometimes fear that they don’t understand machine learning algorithms and its thicket of mystifying terms and concepts. To most lawyers, the mathematics that fuels predictive analytics is impenetrable. These skeptics attack the use of data in part because they simply don’t understand how computers can find patterns in huge amounts of data. The inner workings of the Naïve Bayes algorithm, to pick one of many recondite tools, are a black box. A neural net that takes in wads of facts about a few dozen matters of a similar type and predicts the duration of the next matter—that makes lawyers feel ignorant. John Henry, Esq., loses to a chip. What you don’t understand, you don’t trust and you don’t like. The wellspring of this argument is emotion (fear).
As a final argument, sitting and thinking about a problem feels so comfortable to us that we resist the idea that there are tools that might enable better decisions. No one relies on omphaloskepsis anymore to make choices. Intuition and pondering have gotten society a long way, so why should we rock the boat with data? After all, numbers—as facts—are not neutral, since someone selected them for a reason. Here we find arguments that draw on sociology or philosophy.
Pro Metrics
Now let’s turn to the advocates of evidence-based management. They advance a range of arguments, such as the following five.
Relevant data, mediating between people who hold different values, can bridge value divides. One person commends paralegals as cost-effective, while an opponent condemns them for driving up bills but adding little value. If someone has done a multiple regression on the costs of matters as a function of how many paralegals billed to it, the formula gives disputants a bridge for nearing each other’s beliefs. Data serves a useful purpose as a tiebreaker that can melt hardened views. We can ascribe this argument to philosophy.
Proponents make much of the claim that data ameliorates cognitive biases. Despite boasts that law school teaches lawyers to think step by careful step, all of us remain notoriously prone to distortions in our thinking. As the Nobel Laureate Daniel Kahneman describes so well in “Thinking, Fast and Slow,” the fast cognitive errors that include framing, recency, confirmation bias and salience bedevil our reasoning. Often we are completely unaware of their distortions. Data can counteract blinkered ways of processing information. Data compiled carefully and presented well can challenge prejudices and blind spots. Psychology and neuroscience back this argument.
Those in favor of using data in decisions point out that when numbers are put before people, they can contradict and change beliefs that were thought to be certain and disclose unknown unknowns. Data shakes opinions that are held without factual basis. On the other hand, data can buttress arguments based on other reasoning or rhetoric. Numbers communicated clearly as part of the decision-making process persuade people and provide more management control over the outcome. Rhetoric claims this argument.
Lawyers who amass data can unleash the power of predictive analytics algorithms. In recent years, business has embraced artificial intelligence to glean insights from mountains of data. Lawyers can follow to a degree with their own foothills of data. Additionally, astute general counsel can push their primary law firms to do something useful with the mathematical tools embodied in software programs. Law departments and law firms can use algorithms to make predictions, such as the cost of a case, or they can classify matters, such as which ones are simpler or more complex for purposes of tiered pricing. Algorithms can spot patterns and associations that are inaccessible to lawyers. Mathematics undergirds this argument on behalf of data.
As data analysis makes its way into legal operations decision-making, it can empower younger lawyers. Instead of having to bow silently before “I’ve been doing this for 28 years, so …,” a tech-savvy generation can find and study data, produce charts and push for new thinking. When someone presents unassailable data that tells a convincing story, no one cares about your tenure. Demographics and sociology inform this argument.
What Would Janus Do?
There: a dozen or so arguments for and against data used by legal managers. As senior lawyers learn about and debate the wisdom of how much data should inform decisions, they should always remember that “data doesn't decide; data deepens decisions.” Meanwhile, they would do well to sort out the arguments put forth from both sides, and to appreciate them as stemming from the more fundamental level of disciplines.
Now, let’s move a level higher. Data science for lawyers is part of the industry’s zeitgeist, but since it either threatens or enables change (depending on your views), its advent in law firms and law departments triggers strong disagreements. The preceding paragraphs sketch arguments and link each of them to a broader intellectual discipline. Reverting to the Janus metaphor, we can hypothesize that each discipline provides ammunition for both sides of the debate.
Psychologists would predict the gaming that erodes the value of data but also explain data as antidotes to nonrational thought. Economists might agree on the net present value of data-driven reasoning being cloudy, but they also champion innovation and the accumulation of value by compounding. It seems plausible that the dozen disciplines cited above as a foundation of an argument can each support an argument on the opposite side.
In the end, only good empirical results will grant data a permanent and respected seat at the decision table.
Published March 7, 2017.
