Latest News

Trending: How to Evaluate Legal Technology that Improves Efficiency

This is the first ‘clinic’ from Dera Nevin, who each month will address common questions from readers about the evaluation and implementation of legal technology. Buyers of legal technology can play a critical role in improving the purchase of legal technology by ensuring the strategic questions are asked and fully answered within a business requirements document before going to market, she says.

By Dera Nevin

I dislike much of the marketing around legal technology.  As a long-time evaluator and buyer of legal technology, I find that the information that I am given about products does not help me understand how the technology will likely impact those I buy and implement it for – usually, the working lawyer whether in a law firm or law department – or their business.  I find my observations to be particularly prevalent in the segment of the market that I refer to as “efficiency technology” – products that advertise themselves as helping to streamline legal services delivery.   Let’s dig into what to look at when evaluating efficiency technology and deciding whether it is appropriate to buy, and where and how to implement.

Frequently, the key selling point of an “efficiency technology” is that it improves or eliminates inefficient processes.  Such technology is advertised feature-first, with an emphasis on those features and functions that replace specific manual tasks currently done to deliver legal advice or services.  This go-to-market approach is currently prevalent for expert systems, analytics and “AI” techniques such as natural language processing and machine learning.  The marketing for such technology often emphasises that, once it is incorporated into legal service delivery, it can help with automating manual tasks to reduce time associated with completing work.  The legal technology helps to reduce costs largely by removing billable hours and assigning them to a machine or by permitting the work to be assigned to a lower-cost resource by standardising the work – or both.

Often, and particularly early in an efficiency technology product’s life, the efficiency to be gained is not expressed with precision against a defined benchmark.  The focus of the product’s marketing is on “saved or reduced time” and “reduced cost” without stating explicitly how much time or money will be saved.  Sometimes the cost-savings to be achieved by its introduction are obvious, but this is not always true and the return on investment (RoI) can be difficult to calculate.

While “saving costs” in the delivery of legal services by using technology in lieu of person-hours can be laudable, there is often little analysis or hard fact to show the full scope of the cost impacts from a total-cost-of-ownership perspective (including capital costs and indirect expenses) of introducing this technology.

For example, a lot of the new technology will require significant manpower (often, non-billable) to train and manage the inputs and outputs of the technology; this can be very true of machine learning technology but is not limited to just machine learning technology.  Sometimes these costs are borne by the vendor and incorporated into the cost of the product but, in other contexts, the new users need to train, configure or set-up the technology before it can be reliably used.  This means there can be significant labour and time costs imposed on a purchaser in addition to the cost of licensing or acquiring the software or technology system.

In addition to this labour cost, there can be costs associated with cleaning up data, implementing new workflows or in training before the technology can be relied upon actually to replace the work carried out by humans or permit humans to trust it to use it without human intervention.  During this period there can be significant loss of productivity with hours diverted from billable work, not to mention the need to duplicate efforts (while training or testing the outputs from the technology).  This time and these costs are often not factored into the evaluations of cost-savings possible when “efficiency technology” is used but these are important to factor into any assessment of the cost of the new technology and to evaluate the total cost and RoI of the service delivery associated with its purchase.

Introducing technology to achieve efficiency is still a significant innovation.  There can be challenges to business models associated with the introduction of this kind of technology, as has happened with the movement of much document review to lower-cost and specialty providers.  But efficiency is often really secondary to what has actually happened:  the removal and/or replacement of lawyers from delivery of the work, largely because the technology allowed for standardising an aspect of their work that formerly required opaque judgments that previously could not be easily measured.

Standardisation is valuable, but in evaluating the impact of this technology I also work to understand the total cost of implementing it, because new technology changes the way delivery costs are borne (and by whom).   For example, much of the writing about efficiency technology looks to what costs (often labour) are currently within the service delivery model that will be externalised.  However, there are often new (and sometimes higher) costs that will be introduced alongside the technology that need to be factored into understanding how the technology will impact the business.  In addition to understanding how the technology will impact the business, as I evaluate how the technology performs I also try to understand how the technology will accelerate changes to the business, including the revenue and cost model.  These, largely, are strategic questions and indicate how technology purchases relate to overall business strategy.

Buyers of legal technology can play a critical role in improving the purchase of legal technology by ensuring the strategic questions are asked and fully answered within a business requirements document before going to market.  This has a number of benefits in addition to understanding the true cost of the proposed technology purchase.  Most importantly, when a buyer has really worked through how the proposed new technology will affect the business, more effective implementation decisions may be possible because the question “why is this technology available to me” will be more transparent and relevant to the practicing lawyer.

Dera J Nevin practices Information Governance and eDiscovery at Baker McKenzie LLP and is affiliated with its WhiteSpace Collab Innovation Hub.  Opinions expressed in this article are the author’s alone.  In this space she proposes to address common questions from readers about the evaluation and implementation of legal technology. You can contact the author at  

2 replies on “Trending: How to Evaluate Legal Technology that Improves Efficiency”

Excellent overview of the hidden costs firms need to watch out for. Often quick-to-value, easy-to-learn, incremental improvements within existing tools can deliver better results (faster execution, less stress) than big projects (which have their place of course). One metric that measures improved efficiency is the number of mouse clicks/keystrokes in a workflow – the fewer clicks and screen changes it takes to achieve an end goal, the better – especially if there’s minimal or no training required. At one firm where I worked, the managing partner required a demo of any application that IT proposed to deliver to lawyers – if he noticed any unnecessary clicks, he would refuse approval for rollout until those extra clicks were eliminated. Lately I’ve heard from technologists at a number of firms that they’re intrigued by machine learning possibilities but frustrated at the ramp up time, especially for non-English language document analysis, and still puzzle over how to effectively deliver search or filtered content to business users in the context where they work (MS Word, Excel, PowerPoint, Outlook), without forcing them to leave MS Office or execute extra clicks. They’ve been intrigued by the opportunities opened up by products like Litera Microsystems’ Clause Companion as a minimal-click obvious UI embedded in Word as an add-in, to deliver complex AI-generated content at the point of use (within Word), while also allowing business users to save, contribute, share content organically. So – count the clicks as part of any efficiency project, to make sure it makes life easier. (Full disclosure, I’m a Document Lifecycle Evangelist at Litera Microsystems,

Dera, thanks for this very helpful piece.

I’m a French law librarian and monitoring specialist.

The legal tech and legal AI marketing hype is rampant too here in France. And I can confirm machine learning and deep learning don’t work as well with a non-English language such as French.

From the spring of 2017 onwards, I’ve tried to write, augment and update a long post — a kind of academic paper rather — on legal AI tools available for Frech legal practitioners :

It’s written in French of course but Google Translation do it justice — well, approximately :

Comments are closed.