Last week, a debate erupted in the legal tech community, questioning whether Harvey, a leading AI vendor that sells primarily to law firms, had achieved some of its key metrics around number of users and software utilization (i.e., how many of the lawyers are using the AI tool). Harvey responded by claiming its metrics were better on these fronts than previously claimed. These metrics are impressive and are surely great news for Harvey’s investors who benefit from the sale of seat licenses, but it’s the wrong conversation if we are focused on deploying AI to improve our legal teams. What I really want to know is how is AI creating value for in-house legal teams and the companies they serve?
As John Doerr famously said, “Measure what matters.” Yet in legal tech, the metrics that dominate the conversation are typically those that matter to law firms or venture capital investors—not to general counsels, CFOs, or in-house teams. Retention rates, utilization statistics, and revenue multiples may look impressive for these vendors and law firms, but they don’t answer the questions that matter most to us.
The Metrics We Actually Need
Though retention and utilization rates are helpful directionally to understand whether AI is gaining traction, when it comes to the use of AI to augment legal work, what I really want to know is:
- How many hours of repetitive tasks did our use of AI eliminate (not hours tracked), and how is that reflected in our costs (the total bill, for outside counsel, and internal people cost that can now be redeployed to higher use, for internal teams)?
- How has using AI reduced dependency on outside counsel (not how many seat licenses does the law firm have)?
- How has using AI made us faster and better as a legal team (not feature velocity)?
Today, most legal technology is built for law firms, where more billable hours mean more revenue, and therefore metrics like utilization may be helpful. In-house teams, by contrast, succeed by gaining efficiency, reducing turnaround time, and creating value. We need metrics that measure these goals.
A Case Study: What Metrics Should Matter
I have gone all in on AI—and I believe it will radically transform our legal teams for the better. But instead of buying off-the-shelf tools that we have to figure out how integrate into our workflows (and instead of measuring how many people use those tools), my team is following McKinsey’s advice to find partners and we’re partnering with an AI provider that builds its products and business model from the ground up, specifically to make in-house legal teams better, faster and smarter. Because they only work with in-house teams, we can hold them and ourselves accountable to metrics like cost savings from work that previously went to outside counsel and productivity gains that allow us to redeploy people costs to higher-value opportunities. The metrics we track, which aren’t the number of users or software utilization, are all oriented towards actual measurable value for Coherent.
We hold ourselves accountable to the metrics that create this value:
- Reduction in time to contract (target 75%): We have thus far achieved a 78% reduction in contract review time. Now it’s on to quicker, smarter negotiations.
- Reduction in outside counsel dependency on volume matters (target 60%): we are on track to reduce outside counsel dependency by 50%+ for volume, repetitive matters.
- Creation of in-house capacity created (target 30%): we are on track to use AI to replace 30% of the tasks previously done manually, so we can unlock value in our legal teams and redeploy assets to higher-use activities. As we get better, this number will increase.
These are the metrics that matter to me, right now. They reflect real, measurable progress for in-house teams seeking to become stronger, more efficient and more value add. But they are only the beginning.
Why the Tide Is Turning
Fundamental changes are underway:
- AI is enabling productivity gains that make old business models obsolete. When technology can eliminate 80% of routine work, the billable hour stops making sense.
- AI-native law firms and platforms built for in-house teams are emerging, measuring success in repetitive work eliminated, not work managed our hours billed.
- In-house legal teams are demanding transparency, sharing real numbers and calling out metrics theater.
The conversation is shifting from vendor success to customer value. That changes everything.
A Call for Radical Transparency
To my fellow CLOs: we have an opportunity to shape how AI works for us. Rather than being content to know the number of licenses/seats our outside counsel have bought, or that they are using AI, we should ask all our partners the same types of questions I’m asking Coherent’s legal team:
- How has your use of AI created efficiency gains for our legal team? What specific efficiency gains have other in-house teams achieved with your AI?
- How long did it take to realize those gains?
- What got worse before it got better? What have you learned through failure?
- How does the achievement of massive efficiency affect your business model?
I would love to see a dialogue around the kinds of questions we should be asking to ensure our teams and our partners leverage AI (responsibly, of course) to augment our legal teams and improve our cost structures—specifically when it comes to outside counsel costs, which have increased dramatically (and for certain tasks, are well worth the cost). Share your successes and failures. Build a community of accountability that raises the bar for everyone.
To vendors: Show us the time saved, the costs reduced, and the strategic capacity created. Show us how our incentives are aligned with yours. Share the full journey—including the challenges. Let’s build the future of the legal profession together.
The Path Forward
The legal industry stands at a crossroads. We can continue the AI metrics theater, or we can embrace radical transparency—sharing real data about real transformation.
The question isn’t whether vendors successfully generate revenue. It’s whether we (the customers) succeed in building more efficient, smarter legal teams. Until we build accountability on those questions, we’ll keep having the wrong conversations about legal technology.
Rob Beard joined Coherent in 2024 from Mastercard, where he was chief legal and global affairs officer. Other previous roles include a circa decade-long tenure at Micron Technology, during which time he ultimately served as general counsel and corporate secretary. He has held associate roles at Shearman & Sterling and also Weil, Gotshal & Manges.