After attending UC Hastings College of Law, Richard Dilgren worked in the office of counsel for the U.S. Army Corps of Engineers where he focused on large environmental restoration projects and federal infrastructure construction, developing an expertise understanding the challenges associated with unstructured data and large paper and record volumes. This turned out to be an outstanding foundation for a career as an e-discovery expert, beginning at Evolve Discovery, which became part of what is now known as FRONTEO. He there founded the Discovery Consulting Group, now known as the Data Science Group. This interview has been edited for length and style.
Business intelligence (BI) clearly has had a growing role in both law firms and corporate law departments. What has the evolution of BI been over the last five years, particularly in the litigation and e-discovery context, and who and what have been the drivers of these changes?
Richard Dilgren: It’s an interesting question. The legal field followed most of the corporate world in terms of implementation and understanding the value of BI, both in terms of figuring out how to identify cost centers and how to minimize costs. Corporate law departments have been the drivers of BI adoption, and, in most contexts, the law firms responded to or adapted to fit corporate law department needs and demands.
For lawyers, using BI can be less intuitive. There are certainly larger firms out there that are thinking about these things, but in law there are a variety of different kinds of cases and cost models that make sense, and the spend varies pretty widely from practice area to practice area.
It’s not necessarily true that BI and law are such an easy fit across the board. You need to really evaluate what metrics make sense and figure out how to effectively normalize the data that you’re looking at. It takes a great deal of thought to create and implement a successful BI program at a law firm. I think it’s a little bit easier at a corporation that knows what areas of law they expect to focus on, what their historical spend has been, etc.
Over the past five years, there’s been a significant uptick in business intelligence and sophistication with e-discovery, especially. Before that, it wasn’t really something that people demanded in a structured way. I think that that’s in large part attributable to the increase of business intelligence programs in other areas.
How can corporate legal departments optimize their systems to improve their ability to perform cost benefit analysis of various initiatives, and what tool sets and processes help optimize decisions and performance?
Dilgren: That’s a big question. Frankly, I think it breaks down to a couple of relatively easy steps. The devil’s always going to be in the details, but the very first thing any corporate legal department needs to do is identify what it is that they’re trying to measure. That might sound like rudimentary advice but the reality is that many people start aggregating data without fully understanding what it is that they are trying to understand about that data.
Any useful BI program needs to start from a place where there is a goal identified, or a slate of goals is identified, and then the data that is collected needs to point toward evaluating metrics related to those goals effectively.
That’s step one. Step two is making sure you understand how you want to measure progress. If you’re trying to do an efficiency evaluation for spend, there are a few ways you might approach it. You might look at what your spend is compared to what you expect to settle a case for. If you didn’t expect to settle the case, you might compare what you spend to what you expected that your potential liabilities were. In any case, you need a measuring stick. It’s not only thoughtfulness about the data point collection that’s necessary. It’s also how you are going to measure those consistent data points against a real world outcome.
Obviously the last leg of that tripod is that the data points you collect need to be normalized; so if you’re gathering them from, let’s say, different divisions within the law department, you need to make sure that the data that you’re aggregating is comparing apples to apples. Otherwise, the results that you’re going to get out of your BI program are going to be skewed by the variances in the data that you’ve collected in the front end.
In terms of the tool sets and processes that can help optimize those decisions, I think that’s a pretty big question, also. There are a lot that you can do with nothing more complex than, say, Excel. A lot of people are using Tableau or other data visualizers. We brought in and licensed a graphic visualization tool set for our BI program and built it into our portal.
The easiest way to start implementing a BI program is to partner with other companies like ours that already do structured data collection and already have access to information about your business. Any data point that somebody else collects, as a matter of course and as a part or their routine operations, is something that you don’t have to figure out how to collect on your own.
Our Trust Business Intelligence Portal is an example of a resource that does exactly that. We use Salesforce to manage every aspect of our business from client opportunity identification all the way through identifying what shows up on an invoice.
Clients working with us have access to all that information, and it gives them a jumpstart in figuring out how to implement their own BI program. They can see what data points are already being collected, what those data points are being compared to, how they’re analyzed over time, and things like that. That’s certainly not to say that that’s the only way to implement a BI program. The reality is that, most legal operations at corporations aren’t humongous in size or don’t have the resources to implement a far-reaching BI program without a little bit of assistance from the outside. Leaning on partners who already aggregate data in a structured way is one way to get that started.
How are law departments and law firms changing the way they approach e-discovery and document review, which has historically been one of the drivers of the high cost of litigation?
Dilgren: The biggest systemic change that I’ve observed is a move toward managed services, wherein people are doing structured purchasing of legal services. There are many other things that people are doing in changing the way that they look at e-discovery, shifting the way that their cost models are structured and doing direct licensing and a bunch of other things. Ultimately, using BI programs to understand what your actual service needs and consumption are is the only way that you can really effectively know that you’ve optimized the way that your legal operation is working.
The interesting shift toward managed services has been a relatively painful one, frankly, because the average company, and certainly the average law firm, doesn’t know how much e-discovery service they consume over the course of a year. Different practice groups within law firms have different solutions that work for them. Very often, corporations sometimes operate the same way and sometimes leave it to their outside counsel to identify what services and what vendors they need to use. Figuring out what your actual consumption is seems like it should be much simpler than it usually is.
The ultimate challenge that’s associated with figuring out what you need in terms of e-discovery services is the exact same challenge that’s associated with building and implementing a BI program at its core. The hardest part is making sure that you have data that’s useful for doing the evaluation that you want.
In the case of managed services, that means ascertaining how much data you are regularly collecting and processing. If you’re doing review, how many documents are you pushing out to review and are there ways to save time and money by implementing more efficient processes? Are you heavily leveraging efficient analytics technology or is that a way that you might increase the technology portion of your legal spend a little bit?
There are numerous factors that go into it, but I think that the process of beginning to evaluate what your consumptions rates are is relatively straightforward and of course, there are partners like us, who can help with that. I do think that that’s probably the most dramatic change to how a lot of law firms and law departments are approaching e-discovery, at least viewed through a lens where harnessing business intelligence is really at play.
The second significant change is looking at how heavily people are using text analytics technology, not just linear text but also conceptual analytics technology. The reason that business intelligence comes into play there is that people can see that there isn’t a direct linear relationship between an increase in data volume and an increase in data that needs to be reviewed. In large part the reason is because we’re becoming more effective at eliminating documents that don’t need to be reviewed by humans.
In your current role, what are you seeing is the most meaningful key performance indicators for law departments and law firms, and how can they become actionable?
Dilgren: I think the most meaningful key performance metrics for law departments and for law firms are probably fairly different.
The most significant performance metrics for law departments involve looking at volume of collection and what proportion of the collections are actually being reviewed by attorneys. Looking at how many times each document is actually reviewed by a person, you’d be surprised how much time is spent taking a second look at documents because they aren’t moving through a well structured workflow.
Those are the metrics that law departments need to keep track of in order to structure their purchasing of e-discovery services effectively. They need to know whether the people they’re working with are working efficiently. Obviously, there are other metrics that go into that, and certainly, there are metrics like spend versus risk or exposure that we help with.
What you really need to be able to wrap your arms around initially is: what are you content with? For law firms, they have the luxury of, for the most part, focusing more case by case. Law firms often won’t track things like how much volume is being collected and how much volume is being reviewed across cases because they affect different end clients (corporations or individuals.)
For the law firm, the metrics that I think are the most valuable are the ones that relate to the efficiency of the process. How many documents were eliminated using what technologies, and how does that translate into money saved for their client? How many documents needed to go into a more expensive stage of review, like a privilege review where there’s a little bit more intensive look versus how many were identified as not requiring that scrutiny?
There’s a little bit more nuance because the law firm is focusing on demonstrating that they are effective partners from a technological stand point, and that’s a case-by-case evaluation for the most part. There are other factors that they could consider that relate to implementing a BI program that are internal facing. How efficient is their e-discovery operation? How much data are they processing? If they’re using outside partners, what are the charges and the comparative costs and benefits? How much volume are they putting through analytic and how many people are using it in proportion to the number of people that are engaged in e-discovery?
Those are all things that are more focused on optimizing efficiency of internal operations and less on demonstrating effectiveness to their clients. Most of the time, at least in my experience, when BI programs are implemented at firms, the initial focus is demonstrating that things look clean or that things are efficient to the client to whom they are accountable.
Richard, do you have any case studies or examples of business intelligence success stories you can share with the reader?
Dilgren: We have a large client with whom we have been working for over six years. This corporation has been historically purchasing services at scale but on a case-by-case basis. We had a Master Service Agreement with them, and we were one of only a couple partners with whom they were working. All that essentially did was guarantee them fixed pricing which was good for them, of course, because it guaranteed that even if they had a small engagement they benefited from economies of scale. They received large engagement pricing on their entire portfolio instead of just on their bigger cases, but it wasn’t as cost efficient as it could have been.
The implementation of a BI program, specifically our client-facing business intelligence portal, meant that we started aggregating for them – free of charge – information not only about how much data was going through our pipeline but also how much of that data was being processed, being posted to building case assessment databases, making its way into actual review.
We also began receiving aggregated information about when documents were being used in more than one case so that they could reuse information like privilege. Over the course of a year, we identified that by reusing processed information, and by reusing coding information that’s applied by reviewers, we save them in excess of $2 million. Now, we weren’t doing 100 percent of their work before we performed this analysis, but we took on a much larger proportion after we demonstrated that our BI program was helping them save money and it clearly indicated the effectiveness and efficiency of our process.
Richard Dilgren is a Vice President of Data Science and Strategy for FRONTEO where he focuses on applications of advanced analytics and custom development. He helps clients understand the most advanced technological solutions are available to address their e-discovery challenges and monitor solution progress with business intelligence tools. He can be reached at [email protected].
Published November 29, 2017.
.png )