Editor: Please describe your backgrounds and expertise.
Burke: I am a senior vice president at Epiq Systems, a global provider of technology-enabled solutions for electronic discovery. I work closely with Epiq’s corporate and law firm clients to create and manage high-quality, cost-efficient and defensible solutions during the discovery phase of litigation, including the emerging field of “technology-assisted review.”
I have a legal and business background. At Weil Gotshal, I was a member of the firm’s Complex Commercial and Intellectual Property litigation groups and have litigated cases for a wide variety of clients, including UnitedHealth Group, Reuters, Walmart and H&R Block. As special counsel at Farella Braun + Martel in San Francisco, I had significant litigation and appellate experience. I was also the president of a leading offshore legal services provider.
Benson: I am a director of Consulting Services for Epiq Systems. I have over 25 years of legal industry experience in the areas of strategic planning, client service and project management. I maintain a particular focus on the ongoing challenges of integrating developing technologies with the practice of law. I was responsible for leading Epiq’s initiative in predictive coding and have worked with clients on more than 70 projects involving this technology.
Prior to joining Epiq, my experience included nine years with PricewaterhouseCoopers’ Law Firm Services Group, 10 years as an independent consultant and several years working in a law firm or provider environment.
Editor: Predictive coding (or technology-assisted review) has become a much talked-about subject. Is it all that it’s hyped up to be?
Burke: Discovery has undergone a transformative shift over the past 10 years. We have seen the rapid and explosive implementation of electronic discovery solutions, the adoption of the use of contract attorneys to review documents, and now the emergence of technology as a more significant factor in the entire e-discovery process. These changes have been driven by the sheer size of data that is being collected, processed, reviewed and produced, combined with the realization that document review is an inherently subjective enterprise.
As the Rand Corporation recently reported, document review drives approximately 73 percent of a company’s overall e-discovery spend. Rand opined that the only way document review costs will lessen significantly is through the targeted use of technology. Given the increasing volumes of data being collected, it makes sense to explore all available technologies. However, as Judge Andrew Peck has noted, technology standing alone will never be a “silver bullet” that will obviate the human element. Courts will continue to look at the process surrounding the use of technology. Thus, in the case of technology-assisted review (TAR) software, it is important to create a defensible work flow, working closely with the experts at the company’s provider.
Editor: How have you been using predictive coding technology?
Benson: At Epiq, we were an early adopter of Equivio’s Relevance®. Our clients typically use TAR once search terms have been devised and run against the data, although that is beginning to change. Working with an “expert” (typically a senior associate or partner on the case), Relevance learns what is responsive and scores the document collection, from 100 to 0 (the higher the score, the more likely responsive). Once the documents have been scored, we can then discuss with counsel a proposed methodology for review. Typically, the top-scoring documents will be viewed by outside counsel in the first instance – these are likely to be the most important documents in the case. They are also the type of documents that may contain helpful or damaging information to support counsel’s theory of the case. We can then determine how the remainder of the documents will be reviewed. In addition to scoring the documents, Equivio gives counsel the ability to draw a line at a certain level of responsiveness. Thus, counsel can decide how many documents in the population they want to have reviewed.
When using this technology for the first time, we typically experience a desire to review all documents. However, as we move down the chain from responsive to not responsive and counsel is able to verify the accuracy of the document scoring, we see our clients taking a more aggressive position at the lower levels of responsiveness.
A typical workflow might look like this: the most responsive documents go to outside counsel for their review; the middle tier of documents are reviewed by contract attorneys; and finally, either sample the lower tier of documents, or have them reviewed by less expensive resources. By understanding where the responsive documents are likely to sit, counsel can create a targeted approach to review that has a significant impact on cost.
Editor: How do you decide when to use it?
Burke: To date, we have done over 70 projects using Relevance on a wide array of cases (litigated matters, second requests, incoming productions and internal investigations). Initially, a key concern among counsel was the acceptance of TAR by the judicial community. Now that decisions are beginning to emerge in which courts have agreed to the use of TAR, we find that our clients (both corporate and outside counsel) are looking at TAR much more seriously. Key factors in deciding whether to use TAR are the size of the population and production deadlines. Obviously, some cases are simply too small to justify the use of outside counsel’s time to be the “expert,” although there is no magical cut-off point. Even when counsel are under the gun to produce documents quickly to the government or the opposing party (or need a quick read on the results for an internal investigation), we always advise our clients to think strategically about how they want to review documents. We believe that it is worth the time and cost involved.
Editor: Can you give us an example of the typical ROI or savings you’ve encountered?
Benson: Our clients have seen a range of cost savings as well as experienced strategic advantages by using TAR. It is not unusual for review populations to be reduced by 15 to 50 percent after sampling and verification of the not-responsive portion of a document set. A few projects have seen a reduction of more than 50 percent, and, of course, there is a direct correlation between fewer documents being reviewed and the reduction in costs. In addition, the higher-scored documents, considered to be presumptively responsive, can bypass first-pass review (and the related costs) and move directly to second-pass/privilege review.
Another area for cost savings is in the review of incoming productions. Many of our clients enjoy tremendous cost benefits by using Relevance, which is trained in these instances to identify “important” material. The cost savings of not having to review all produced documents is significant, and the time value of providing the case team with those documents that are important soon after receiving the production is tremendous.
Editor: Is the use of predictive coding just an ROI decision, or are there other considerations at stake?
Burke: Economics obviously is a key driver in the development and interest in TAR. As the Rand study reported, technology is the only way to bring down the cost of document review to levels that are proportional to collection and processing. However, the prioritization of documents through the use of TAR can have several critical substantive benefits as well. For one thing, counsel will become aware of the key documents in the case quickly. These are the documents that will likely be admitted at trial, form the basis of a defense or even influence the settlement thought process. Furthermore, from a risk management standpoint, having outside counsel serve as the “expert” allows the law firm to make decisions that will shape the entire review. Finally, when combined with other tools like “email threading” and “near dupes,” the use of technology drives greater consistency in the tagging of documents. Thus, while likely the prime driver, ROI is but one of several important variables in the TAR equation.
Editor: Which predictive coding system do you use, and what are the key factors in your decision to use this system?
Benson: We have worked with Equivio and its Relevance tool for close to four years, and our clients have had terrific results. In evaluating tools, we looked to several factors, such as the following:
Transparency of results: Attorneys should have the opportunity to have a clear understanding of the results as well as to evaluate and validate those results. We wanted to work with tools that provided graduated scores, not a binary decision of responsive or not responsive. We felt the ultimate decision about responsiveness should lie with the case team, not an application.
Metrics to evaluate process and results: It is important that the learning process as well as the results be supported by statistically reliable metrics.
Ease of use from the lawyer’s perspective: Since the expert who is training the application also has other responsibilities, it is important that the technology be easy to use and allows the attorney to work as independently as he or she can.
Editor: How has the role of the e-discovery service provider changed with the advent of predictive coding projects? Would it be fair to say that the consulting component of the service provider’s offering has become more important?
Burke: Judge Peck noted that the most important element when using TAR is the process that surrounds the technology. Technology is not an end in itself. Given that fact, the role of the service provider takes on a new complexion. As counsel is working with its provider to ready data for review, he or she should carefully consider how best to use technology. As one example, the government encourages lawyers to get their service providers involved in early discussions about the process surrounding the use of e-discovery technology. At Epiq, we are seeing a growing importance of the role of the consultant and are gearing our teams to be able to meet those needs.
Editor: How do you measure success? How can users know whether the software is doing the job it’s intended to do?
Benson: Success is measured by a combination of factors, all of which contribute to the level of comfort and satisfaction that the client feels with the resulting production and case preparation process. Since the goal of e-discovery is to find and produce the relevant/responsive material, it is critical that the attorneys have confidence both in the technology used as well as the protocols and workflows that support the overall process.
Published October 20, 2012.
