Surveying the Surveys: There are many in the legal world, and the quality is uneven

Based on my research, during the past five years at least 90 different U.S. organizations published reports based on 190 surveys of U.S. law firms or law departments. That plethora of legal-industry surveys addressed a wide swath of management data. An analysis of the topics finds that compensation, e-discovery and outside counsel cost control were frequent topics, but all manner of other data inquiries were also carried out. The sponsors were primarily publishers, vendors of software or services, bar associations and consultants. At least half a dozen law firms and several trade groups also launched surveys.

In any given year, several thousand law firms or law departments take the time to complete one or more of these instruments. Multiples of that received an email invitation to participate. And I’m only talking about legal-industry surveys that produce reports available to participants and the public at no charge. Other surveys produce proprietary reports for their clients or to be sold to the public.

During 2017 the pace of these surveys continued to swell. General counsel and managing partners may be suffering from survey fatigue (the largest firms and law departments certainly are). I envision no cure for this problem, since every sponsor seeks to collect data for their own purposes.

Given the significant investment of time, attention and money dedicated to the collection of management-related metrics, it is instructive to step back and consider the advantages of all this effort – as well as its drawbacks. Let’s look first at what is admirable about legal-industry surveys, then turn to criticism of the survey surfeit.

The Benefits

Surveys and reports benefit law firms and law departments in at least six ways.

1. They create aggregated insights from isolated data. Surveys epitomize the magical process that transmutes individual bits of data into valuable aggregated information. A beneficial “Comedy of the Commons” (the more people donate data, the more valuable the collection becomes), everyone benefits if many people contribute their private data to a survey. No one can find out how many paralegals the law departments of utilities companies have unless someone asks them and they tell you (or a government agency requires the data to be compiled). Then everyone can make their management decisions knowing that the median is one paralegal for every 4.3 lawyers.
2. They provide empirical content. Another benefit of survey reports is the light they shed on a topic. Knowledge is power and quantitative knowledge even more so. A survey might uncover the average compensation of senior partners at firms having more than 500 lawyers or the median number of lawyers in financial services for every billion dollars of revenue. Without some empirical basis for making a decision, leaders of lawyers flounder along with intuition and anecdotal impressions. With survey data, they can think and make decisions more sure-footedly.
3. They provoke discussion. Give legal industry surveys credit for promoting debate. If a counterintuitive metric turns up as a result of a survey, articles and conference presenters and consultants can chew over the causes and consequences. If reliable data supports a position – convergence of law firms doesn’t save money, leverage doesn’t reduce total legal spend, practice groups don’t need to achieve critical mass to be profitable – management acumen will increase. Apt data sharpens arguments.
4. They highlight trends. Surveyors like to show trends in data over time. They want to indicate a direction of change for some metric and make predictions based on it. For instance, has total legal spending as a percentage of revenue been declining or increasing in an area of law? Has the average realized billing rate of large law firms increased faster than inflation since 2008? If a survey organization asks the same question more than one year in a row, data trends show up. Or sometimes a one-shot survey asks for this year’s figure and last year’s figure. Once you see a temporal pattern, you can speculate more credibly on the forces of change.
5. They promote management metrics. Surveys build awareness of the value of data analytics. Inevitably a survey discusses averages and medians and maybe ventures into such descriptive statistics as standard deviations or interquartile ranges. Every now and then a trend line hints at regression techniques behind the scenes. When surveys explain smoothed lines, outliers or contingency tables, readers at least subliminally come to appreciate how data can be woven into wisdom.
6. They introduce graphical techniques. Almost all survey reports display their findings with tables, plots or infographics, so they promote better understanding and use of powerful visualization tools. A scatter plot, for example, presents all of the data for two variables, enabling our eyes to pick out patterns and conclusions. Lawyers’ ability to “read” a sophisticated plot may lag their ability to read a regulation, but surveys that visualize data effectively proclaim the power of showing a story with a picture.

The Limitations

No less than six forces for good, yet all is not cheerful and bright in the world of legal surveys. Consider these five concerns.

1. They offer limited transparency. Almost no legal surveyor adequately discloses how they went about their work. A lack of transparency pervades the source of invitees, the percentage of respondents, selection biases and much about the cleaning of the data. The tenets of reproducible research that prevail in scientific publications have barely penetrated our industry’s surveys. No one posts their data set for the world to see and work with, unlike many scientific researchers. Before anyone relies on reported findings, they ought to be able to judge the objectivity, competence and diligence of the sponsor who published the findings. Those assessments of credibility require much more disclosure.
2. Their methodology is rarely disclosed. A person reading a survey report finds it almost impossible to understand the methodological choices made by the survey preparer. For example, did they impute any numbers where survey respondents failed to provide data? Did they handle anomalous data in a responsible fashion? Did they remove duplicate submissions or spurious responses? Compounding both the lack of transparency and mysterious methodologies, there is no organization in the industry that audits surveys and their reports. Data is collected, a report is published, and no one can be the wiser about the many decisions made along the way. No one does Rotten Tomatoes for legal industry surveys.
3. They are often self-serving. Nearly all surveys are carried out by an organization that has a stake in the findings. Why else would they invest their time, talent and dollars? An e-discovery company doesn’t mind publishing data about the frequency of document-intensive lawsuits. A matter management vendor favors metrics that outside counsel spending is rising. Search firms thrive when compensation packages vary widely. The inevitable biases that flow from trying to make money infect surveys in our industry. On the other hand, the benefits of most surveys come at no cost (especially to nonrespondents), and any reader can presumably take into account the commercial interests of sponsors.
4. The data is siloed. No one has yet found a way to combine data from more than one sponsor’s survey. If it were possible to combine data sets, richer insights could be generated. (This would obviously be easiest to arrange with data from the same companies or law firms.) Perhaps even proprietary data could be cleansed and contributed to a common repository.
5. They are illustrated with unimaginative graphs. A particular bugaboo of mine is that the quality of graphics in survey reports falls far short of what could and should be crafted. A boring succession of overly decorated bar and pie charts numbs the reader. With software that can create elegant graphics so plentiful, it is a shame that tired 20^th century visuals dominate survey reports. Many techniques that would sharpen them are inexplicably overlooked.

The Bottom Line

The more data we have about legal operations and the management issues arising from them, the more effective and efficient lawyers can be. As legal data analytics gain prominence, we can project that even more surveys will follow. Everyone wants data. And as our profession strives to become more efficient, it badly needs reliable and pragmatic information. When surveys are well crafted, they stand as the best way to obtain the knowledge we seek.

You may also want to see my previous article on this subject that appeared in the September 2015 issue of Metropolitan Corporate Counsel: Should Law Departments Rely on Survey Findings?

Rees Morrison is a partner at Altman Weil, Inc. He focuses on law department consulting and data analytics for lawyers. He recently published a book on LeanPub, “Data Graphs for Legal Management.” He can be reached at [email protected].