The Legal IT Innovators Group (LITIG) led by CMS’ chief innovation and knowledge officer John Craske has launched a benchmarking initiative that aims to agree an industry-wide methodology for measuring the output of generative AI technologies.
The Legal Industry AI Benchmarking Collaboration will gather views from suppliers, law firms and academia around how to ensure a sufficient level of transparency and standardisation among suppliers and tools. If you want to be consider for inclusion in a cross-industry group to create an open source benchmark for testing generative AI click HERE.
Speaking to Legal IT Insider, Craske said: “There needs to be more transparency. I see it in same way as when you buy a new car, you get a miles-per-gallon figure based on a defined set of real world circumstances. We need something similar for generative AI in legal so that we can all measure and trust figures that are produced. At the moment there is no transparency, no rule or mechanics.”
Craske added: “I’m not saying that just because generative AI isn’t perfect, it’s not useful. We have seen some brilliant use cases and are committed to driving adoption. But if there’s no transparency then it’s just marketing. We need to be able to compare suppliers, be able to see and measure developments and improvements over time and build trust with suppliers and amongst our lawyers and clients.”
LITIG is the UK’s leading legal IT industry body and its members represent around 90,000 users and £20bn of turnover.
Asked how LITIG will respond to any potential suggestions from suppliers that they can’t be expected to give away competitive secrets, Craske said: “We’re not asking how you got to where you got. It’s about testing the products available to customers and testing the output and how that compares with what is promised.”
Craske is encouraging people who are able to contribute meaningfully to the conversation to sign up, and again that link is HERE
Both in-person and virtual meetings will be held to brainstorm a path forward and we look forward to seeing some of you there and lending our support to the initiative.
See also:
Leading legal IT industry body launches due diligence questionnaire for suppliers of AI
https://forms.microsoft.com/pages/responsepage.aspx?id=l7Laja8Rdk-3BMGKHStwL7QDn6KrJ59GsUu-EGgbnlBUQVFEUlpMQTExU1BIUjZHNDdDODhBRThBMi4u