Van Halen, M&Ms & e-Discovery Metrics

Topics: Corporate Legal, ediscovery, Law Firms


Effective metrics can help to ensure both quality and efficiency in legal processes. One area in particular where metrics can make a big difference is in the process of searching, reviewing and producing documents in litigation and in investigations.

Making the search and review process more efficient and effective can make a big difference, particularly considering the outsize impact it has on overall litigation costs—70% of discovery costs are spent on document review, according to Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery (RAND 2012).

A White Paper that David Grant of FTI Technology and I recently authored, entitled Metrics that Matter: Van Halen, M&Ms and Measurement in E-Discovery, discusses several metrics that can be used to help ensure the quality and the cost-effectiveness of the search-and-review process.

You might ask what Van Halen and M&Ms have to do with measurement in e-discovery. Well, the band cleverly devised a strategy involving M&Ms that enabled them to quickly assess whether those responsible for installing their elaborate stage and sound system in each concert venue had followed their instructions.

Buried deep in the band’s long list of instructions in its 53-page tour rider, Van Halen required a bowl of M&Ms backstage, but with one important caveat: All of the brown M&Ms had to be removed. Failure to comply would result in the concert promoter’s forfeiture of the entire show at full pay.

If the band arrived on the day of the concert and found brown M&Ms backstage, they knew that their instructions weren’t followed closely. So they would conduct a thorough quality review of every step of the stage setup. If the brown M&Ms were removed, as instructed, the band had greater comfort that their instructions had been followed, and they could focus on testing a few key pieces of equipment.

Just as the colored M&Ms provided a quick visual metric of quality, several measurements can be used to test the effectiveness of search-and-review.

Prevalence: Prevalence measures the proportion of the document population that has a particular characteristic—most commonly the percentage of documents likely to be relevant to the issues in the litigation. Knowing Prevalence not only enables planning for the right resources and employing the best search-and-review strategy, but it also can help you establish your “goal” for retrieval processes. (For example, how many responsive documents exist that your keywords, technology-assisted review process or reviewers are trying to find).

Recall: Recall is the percentage of the total responsive documents in a document population that a search or review process actually finds. Recall is probably the most important search-and-review metric, as it allows you to evaluate the performance of search terms, document review, and technology-assisted review processes such as predictive coding.

Precision: Precision measures how well a search or review process yields only responsive documents. Having search terms or a predictive coding process that yields high levels of Precision is key to avoiding the inefficiencies and high costs of reviewing large numbers of irrelevant and nonresponsive documents.

Depth for Recall: Depth for Recall measures the proportion of the document population that you must review using a particular search process to achieve a given Recall level. Depth for Recall is important because it indicates the amount of work that a particular search-and-review process will involve, and allows comparisons between different methodologies.

In sum, a few metrics can make a world of difference in helping you to develop an efficient and cost-effective search-and-review process, and to defend the ultimate results upon making your document production.