Guest article by Jay Leib, Chief Strategy Officer & head of the advice@kCura team
Computer-assisted review uses analytics technology to leverage the expertise of humans and suggest coding decisions in a document universe. The goal is to help case teams prioritize documents, sort out large groups of non-responsive documents, and ultimately facilitate a faster, more cost-effective review without sacrificing quality.
Understanding how the computer-assisted review process works—and knowing best practices for working with the technology behind it—can improve your results tremendously. To aid folks who may be thinking about using computer-assisted review, we’ve thought of seven expectations that should be carefully considered before and during the process.
(1) Clearly define why you plan on using the technology, and what a successful outcome would be for you.
Computer-assisted review is a technology enabler and, as such, it must meet the strategy and objectives of the review team to be successful. All litigation technology– or all technology for that matter – will be more appropriate in some scenarios than others. Prior to starting a computer-assisted review, the review team should define the strategy of the review and the goal of the technology.
Some of these goals may include:
- Prioritization, or quickly categorizing documents by relevance or issue for batching
- Removing the noise in the form of non-relevant docs, while teams review the relevant ones
- Quality control, or comparing the results of computer-assisted review with human review and seeing if there are any discrepancies, in a surgical manner
(2) Understand that it takes time to build a proper index.
A well-constructed analytics index is an essential prerequisite for every computer-assisted review project. It is important to put effective filters on the index so it excludes any text in the documents that is not conceptually relevant. This step requires time for planning and time for the engine to do its work; however, exactly how much time is needed will vary from project to project. Allow sufficient time in your project plan for an analytics index to be built between processing and computer-assisted review.
(3) Understand that some documents will need to be manually reviewed.
Some documents – handwritten scanned images, graphics, and documents containing only numeric data – cannot be indexed by a text analytics engine because they have no conceptual meaning for the engine to use. This means that not every document should go through the computer-assisted review process.
(4) It’s crucial to have the right domain experts from the start.
Domain experts – those familiar with the intricacies of a particular case and the type of case at hand – should be involved in computer-assisted review from the start. Without a domain expert to train the system with good example documents, your computer-assisted review may not achieve the desired results. The computer-assisted review engine assumes all documents coded and quality controlled by humans are “the truth.” This is why it is imperative to utilize reviewers that have a solid understanding of what is relevant for a specific case.
(5) Your domain experts must be trained on the specifics of reviewing documents for a computer-assisted review workflow.
It is important to teach everyone involved in the computer-assisted review project best practices for teaching the system through training documents. The best way to learn about what makes a good training document is to understand a few rules about what makes a poor training document:
- A document should never be included as an example based on the content of a family member.
- A document is only a good example if there is text on the document’s face. If there is no relevant text within the four corners of the document, it is not a good example document.
- If numbers or symbols make a document responsive, or if a spreadsheet contains predominately numbers – as opposed to text – then that document does not make a good example.
- A document that generally has poor-quality OCR, such as a handwritten document, does not make a good training document.
- Documents that are highly formatted and contain small amounts of text – such as schematics, engineering drawings, etc – might be responsive, but make poor training documents.
Reviewers need to understand that the quality of the training document is independent of the document’s relevance. A reviewer is making a decision on both responsiveness and whether or not a document is a good example to use to train the analytics engine.
(6) The results of computer-assisted review must be validated.
After establishing domain experts and preparing the analytics index, it is important to use statistical sampling to gauge the effectiveness of the review protocol. Sampling the results of the document universe is how you verify the system is achieving the criteria established by the review team. Statistical sampling will also provide valuable insights that may supply the analytics engine with additional training documents or suggest a course correction to the review protocol.
(7) If reviewers disagree, it will confuse the system and potentially cause high turnover rates. Make sure you take steps to track and review disagreements.
A single reviewer driving the computer-assisted review process would be ideal, but this isn’t always possible depending on your timeline. If you have multiple domain experts working through the computer-assisted review training and quality control rounds, having them on the same page and establishing a process for questionable documents can mitigate confusion as to what defines a responsive document. Reviewers should keep notes explaining why they coded a training document the way they did, so they can be referenced later. Consistency is key, and you’ll need to be prepared to address inconsistency through your quality control process. If a reviewer is not confident in how a document should be coded, then that document should not be utilized as a training document.
It takes a well-trained team to drive the computer-assisted review process and ensure it is effective and defensible. Understanding how the process works and learning best practices for working with the technology is key.