Trainable E-Discovery Software Offers Cost Savings

Tuesday, January 5, 2010 - 01:00

The Editor interviews Chris Dale, commentator on e-discovery.

Editor: Describe what you do.

Dale: I am a lawyer turned litigation software developer turned commentator. I was a litigation partner in London and then spent 13 years developing and marketing litigation software and working with lawyers. I now run the e-Disclosure Information Project, which brings my knowledge and experience to helping lawyers, suppliers, judges and clients who learn how proper use of both rules and software applications can reduce the cost of e-discovery. Equivio is one of the sponsors of the Project.

Editor: In the U.S. and UK, one of the key challenges facing litigators is the cost of review. Lawyers have been brought up to read every document. Is there a valid, defensible alternative?

Dale: Over the past decade there has been vast growth in the number of documents, and particularly emails, which may have some relevance to litigation, and the lawyers have a duty to produce that which is relevant. Americans seem to produce many more documents per case than we do in the UK, but, although the scale is not so large nor is the quantity of litigation so great, we have a similar problem in the UK.

What is discoverable in U.S. litigation is everything relevant to a request made by the other party, and relevance is an extremely broad test. The UK is slightly different in that we must produce only those things that advance our case or are adverse to the case of the disclosing party or of any other party.

At the outset of litigation, and quite apart from the formal obligations of discovery, you need to advise clients very quickly whether they are likely to win or lose, which in many cases is going to be determined by the documents. The newer generation of software enables you to distil what is important from the mass of documents and to estimate what the litigation is likely to cost. Clearly, the goal is to diminish that pile, and to do so in a defensible manner, before the expensive review stage.

Editor: Does technology exist that can reduce the pile of documents that need to be reviewed by lawyers and thus make it possible through early case assessment to get an early fix on the outcome of the case and the cost of e-discovery?

Dale: Indeed there is. It's a challenge, because in order to address this effectively, the software needs to cross the chasm of organizing data to analyzing content. Equivio>Relevance is one of the new tools that is making this almost magical leap.

Although there is a natural human tendency to defer hard decisions, the rules in both the U.S. and UK militate against that. In the U.S., for example, you have an obligation to meet and confer within 120 days after the complaint has been served. Parties in the UK have an obligation to discuss their sources as a preliminary to an order for disclosure. The litigation process in both jurisdictions is a two-step process. The first step is to establish what document sources the client has and to get a fix on the case and e-discovery costs. The second is to refine and review the documents for production. Early case assessment software and document review software splits fairly neatly into those same divisions - although Equivio>Relevance can be used in both steps in the sense that early decisions as to broad relevance may be reappraised as more becomes known about the case and the documents.

Lawyers brought up the old way can be uneasy at the thought of delegating any of this to machines, partly because in the U.S. that won't help them avoid sanctions when the other side challenges their discovery. Some of the U.S. cases emphasize the importance of showing what you have done - to be able to provide an audit of your decision making.

The reason why Equivio's new product is called Relevance is that this is what it does - it determines the relevance ranking of every document in the collection. The interesting thing about it is that it does so in a way that involves the lawyers heavily, so they are not simply delegating that function to a machine, but continuously interacting with it in a way that gives them comfort that they are actively engaged in the process.

Editor: Tell us briefly what Equivio>Relevance does?

Dale: All Equivio products have names that state precisely what they do - it is obvious what a product called Equivio>NearDuplicates or Equivio> EmailThreads is for. Equivio>Relevance identifies, say, 50 documents for lawyers to review and code according to the lawyers' estimate of their relevance. The documents then go back into Equivio>Relevance which generates another 50 documents. The relevance scoring is progressively improved as Equivio>Relevance learns from each sample.

After perhaps 35 or 50 iterations of this process, Equivio>Relevance decides that it's got enough information to make coding decisions about the rest of the population and very quickly (and speed is a vital element in this context) ranks them and puts them into piles according to their relevance, distinguishing between those which must be reviewed, those which might warrant review and those which can be left on one side, for now at least. Senior lawyers can be set to reviewing the highly relevant pile, with other documents given a lesser priority in terms of lawyer seniority and urgency. A decision to leave low-priority documents on one side can be reconsidered - all the documents remain available in case subsequent decisions or the result of the Q&A process cause a re-appraisal of the relevance criteria. This latter element - the ability to take account of changes in the case, and feed the results back into the process - is critical to lawyer acceptance of computerized relevance ranking. Exercises like this involve managing cost and risk - over-inclusion results in the review of too many documents; under-inclusion brings the danger of missing key documents. The system allows you to see how many documents fall within each category, and the user can adjust the relevance threshold to bring more or fewer documents within the definition of "relevant."

You can export that relevance ranking data into a review application - it is added to all the other information stored about a document such as date, sender, recipient, custodian, etc. The capture of the relevance ranking means that your searches can include the fact that Equivio>Relevance has ranked the document as highly relevant.

Editor: You differentiate between function and process?

Dale: Yes, the functional element is what Equivio>Relevance does. Its purpose is to sift the relevant from the irrelevant and to prioritize those documents in terms of relevance. However, the technology also assesses the quality of the information it generates.

This process of interaction of lawyers and Equivio>Relevance provides auditable documentation of the trainer's decisions, and a view of how the relevance score was calculated for each document.

Editor: To what extent are the results obtained from Equivio>Relevance dependent on the choices made by the lawyers with respect to the documents reviewed by it and their coding in the training process you described.

Dale: The results are dependent on the quality of the human input. So it's important that the training process be consistent and of high quality. The "trainer" should be a skilled lawyer, thoroughly familiar with the case.

Editor: Many lawyers feel it their duty to read every document. Is that not what is in fact required of them?

Dale: You can't read every document by eye if you've got hundreds of thousands of them; neither the deadlines nor the economics allow it, and you have to have some means of filtering it down. One of the misunderstandings is that what you're doing is delegating the whole thing to Equivio>Relevance. You are not. What you are doing with it is prioritizing, working iteratively with Equivio>Relevance to decide where to look first.

Lawyers cling to the alleged "gold standard" of human review because they were brought up to work that way. You can only test the relative accuracy of two approaches by doing the exercise at least twice over the same document populations using different methods. This has in fact been done both in the TREC (Text Retrieval Conference) 2008 and for a case study done jointly between Equivio and Epiq Systems (whose review platform DocuMatrix incorporates Equivio>Relevance). In the Epiq test, Equivio>Relevance and a human review team agreed as to 91.4 percent of the relevant/not relevant choices, and a human "oracle" sampling the disputed documents sided more with Equivio>Relevance than with the humans.

The lawyer participation in this is vital, both when the initial relevance decisions are being made and in the Q&A stage. The combination of the lawyers' skill and the system's accuracy brings consistency, which cannot be expected of the lawyers alone (not in any sense which is economically viable anyway), an auditable result and speed. Lawyers are not compromising by using tools like this in place of purely manual review.

Editor: What are some of the ways that in-house counsel use Equivio> Relevance?

Dale: In-house counsel use Equivio> Relevance for corporate investigations and in early case assessment for litigation. In both these situations, you need to focus on the most relevant documents in order to make informed decisions. In litigation, the challenge is to decide whether to settle or defend. To make this decision, you need to identify the key issues and the level of risk, to understand your side's strengths and vulnerability, and to make an estimate of the cost of pursuing litigation. This is largely a function of the number of documents that will ultimately need to be reviewed. Corporate counsel are increasingly taking these early stages of discovery in-house. We see corporate counsel adopting and building in-house expertise in the use of tools like Equivio>Relevance. The aim is to respond more rapidly and reduce cost by lowering dependence on outside counsel to perform these key functions.

Please email the author at with questions about this article.