In re Actos: Ten Features Add Up to a Strong Predictive Coding Protocol

Jan 8

2013

In re Actos: Ten Features Add Up to a Strong Predictive Coding Protocol

By Cynthia Courtney, Esq., VP, Discovery Engineering

Whether you call it technology assisted review (TAR), computer assisted review (CAR), or predictive coding, 2012 was a blockbuster year for advanced search applications (we will call it TAR in this post). The year started off strong for TAR at Legal Tech/New York, with bloggers commenting on the buzz about the new technology. Shortly thereafter, a drama unfolded in Da Silva Moore v. Publicis Group, an employment case in which the parties were negotiating a predictive coding protocol. In his February 24, 2012 opinion, U.S. Magistrate Judge Andrew Peck made clear that parties using predictive coding must design an “appropriate process” that includes “appropriate quality control testing.” An unusual development ensued in which plaintiffs challenged the protocol through a request that Magistrate Judge Andrew Peck recuse himself because he had appeared on a Legal Tech panel with a lawyer from defendants’ legal team. The district Judge, Andrew Carter, upheld Judge Peck’s decision not to recuse himself, approving the protocol’s “standards for measuring the reliability of the process and the … buil[t] in levels of participation by Plaintiffs….the search methods will be carefully crafted and tested for quality assurance, with Plaintiffs participating in their implementation.” Judge Carter concluded that to call the proposed TAR method unreliable at that stage of the proceedings would be speculative.

The principles of cooperation, transparency and attention to a strong sampling workflow emphasized in DaSilva Moore have a concrete and defined role in In re Actos Products Liability Litigation, MDL No. 6:11-md-2299 (July 27, 2012). In that case, the parties entered into a detailed agreement to use Equivio Relevance. The parties included the following features in their protocol:

1. The parties first delineated key custodians as well as sources of defendants’ ESI (including a number of databases) and agreed to apply predictive coding to a “sample collection population” of four custodians’ e-mail.

2. Each party appointed three subject matter experts (SMEs) to jointly review documents in order to make a relevant/non-relevant determination. If at any point in the process the SMEs could not agree on a relevance determination, the parties were required to meet and confer to achieve agreement before contacting the court for resolution if they could not agree.

3. In the “assessment” phase, the initial random sample of 500 documents would be drawn and defendants’ SME alone would review them for privilege. Privileged documents could be withheld entirely or redacted and shown to the other party’s SMEs, but in any event could be left in the set that would be used to train the system.

4. Using the statistical metrics generated by the TAR system, the parties agreed to continue the assessment phase until the initial set of documents contained a specified number that were relevant.

5. Both parties’ SMEs would then review the sample set and make a joint decision regarding relevance. These jointly coded documents would be used to train the TAR system.

6. The parties agreed to meet and confer to determine “which relevance score will provide a cutoff for documents to be manually reviewed by defense counsel for production”—that is, if the parties agree that enough documents will be reviewed to yield 75% of the relevant documents, the cutoff represents the percentage of the universe of documents that must be reviewed.

7. In the “iterative training phase,” the parties’ SMEs would review samples of 40 documents each until the system reaches a stable training status.

8. Prior to generating the review set, the parties would “Test the Rest” to perform quality control—that is, to sample the documents whose relevance score is below the agreed-upon cutoff and assure that the statistical assumptions upon which the review was based were sound.

9. After the review but before the production was to be generated, the parties agreed to sample and collaboratively review documents that were above the agreed-upon relevance cutoff, but were not deemed production-worthy by defendant.

10. Plaintiffs agreed to bear their own costs of compliance with the protocol (a form of cost-sharing, given that parties under the Federal Rules of Civil Procedure are generally responsible for bearing their own production costs.)

This protocol is a solid road map for parties seeking to use TAR systems, for three reasons.

First, it is transparent. It allows the non-producing party to have a say in what documents are considered relevant in the universe of documents, which is more than non-producing parties traditionally have had. Second, it requires cooperation. At each step, both parties must be satisfied with the results before moving forward, and can apply to the court for relief as a last resort. Finally, it reflects a strong sampling and feedback workflow, which lends credibility to the results.