D4 eDiscovery Service Blog
Nov 14

By Chuck Kellner, SVP, Discovery Engineering

"Test the Rest" - Statistical sampling in review, order from Judge Grimm“Test the Rest” is a phrase coming into popular usage in eDiscovery. It means that for whatever part of your collection you are not going to review for potentially responsive material, you should test it with statistical sampling.

A case for “Test the Rest” lies in Judge Grimm’s criticism of the party who wanted to rely on mere searching as a basis to justify clawback of erroneously produced privileged documents.

“Common sense suggests that even a properly designed and executed keyword search may prove to be over-inclusive or under-inclusive, resulting in the identification of documents as privileged which are not, and non-privileged which, in fact, are. The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive.”†

What does Judge Grimm say here? At least sample the documents you have searched and for which you intend to claim a privilege. And, at the very least, TEST THE REST!

Finally, and most gratifyingly, we can rely on the weight of Judge Peck’s primary decision on predictive coding.

Important for “Test the Rest” are the following passages:

“In the final sample of documents deemed irrelevant, are any relevant documents found that are “hot,” “smoking gun” documents (i.e., highly relevant)? Or are the only relevant documents more of the same thing? One hot document may require the software to be re-trained (or some other search method employed), while several documents that really do not add anything to the case might not matter.

“Quality Control by Random Sample of Irrelevant Documents. In addition, at the conclusion of this search protocol development process described above, and before the final search and production described in Paragraph 7 above, [the producing party] will review a random sample of 2,399 documents contained in the remainder of the database that were excluded as irrelevant.”‡

We are being told here that a “Test the Rest” procedure is what validates the use of the new technologies.

Not just by implication, and to no surprise, the exact phrase “TEST THE REST” shows up literally in a judge’s signed order:

“9. Quality Control By Random Sample of Irrelevant Documents. [T]he parties…will review… a random sample of documents in the sample collection population with relevance scores below the cut-off score set for establishing the review set (aka the “Rest”). These documents are flagged for culling and will not be included in the review set. In Equivio Relevance, this test is referred to as “Test the Rest”. The purpose for this phase is to verify that the Rest contains a low prevalence of relevant documents and that the proportionality assumptions underlying the cut-off decision are valid.”¿

What does that mean? After the predictive coding (using Equivio Relevance) and after the review of the relevant documents down to a particular relevance score, the parties will TEST THE REST to make sure they didn’t leave behind anything important.


You can look at more cases than this to tell you to “Test the Rest”§. Coming up with a search and sampling protocol is pretty easy. Try out your search terms. Look at the report of the results. Look at samples of the results. Adjust your terms to take into account under-inclusion (missing responsive) or over-inclusion (false search hits). Repeat. Then sample what is not going to be included in your review.

Whether you select your review set by keyword search or predictive coding, no matter what technology you use, if you still want only to guess at how effectively you are culling or reviewing, you should be prepared for a challenge to your production.


Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008) Hon. Paul W. Grimm, United States District Judge, D. Md.

Da Silva Moore, et al. v. Publicis Groupe & MSL Group, U.S.D.C.,S.D.N.Y. Case 1:11-cv-01279-ALC-AJP Document 96 Filed February 24, 2012. Hon. Andrew J. Peck, United States Magistrate Judge

¿Case Management Order In Re: Actos (Pioglitazone) Products Liability Litigation USDC WD La, MDL No 6:11-md-2299 July 27, 2012 (Hon. Rebecca F. Doherty, United States District Judge; Hon. Patrick J. Hanna, Magistrate Judge

See also Greg Buckles’ “Actos Case TAR Protocol Order – Equivio’s Relevance in Action?

§Kleen Prods. LLC v. Packaging Corp. of Am., No. 10 C 5711, 2012 WL 4498465 (N.D. Ill. Sept. 28, 2012) ; Nat’l Day Laborer Org. Network v. United States Immigration & Customs Enforcement Agency, — F. Supp. 2d —, 2012 WL 2878130 (S.D.N.Y. July 13, 2012); Mt. Hawley Ins. Co. v. Felman Prod., Inc., 2010 WL 1990555 (S.D. W. Va. May 18, 2010); ClearOne Communications, Inc. v. Chiang, 2008 WL 920336 (D. Utah Apr. 1, 2008)

Tags: , , , , , , , , , , ,

Leave a Reply


Connect with D4