A law professor at the University of British Columbia (UBC) has concluded that Lexis+ AI’s content is “riddled with mistakes” and should not be used by law students yet, following a series of tests of the generative AI research and drafting solution.
Benjamin Perrin, who in addition to being a legal professor also leads the UBC Artificial & Criminal Justice Initiative, gave Lexis+AI multiple prompts, which he has shared in depth alongside an article for the Canadian Bar Association’s National Magazine.
Lexis+ AI is a generative AI-powered legal assistant that was first released in the United States in May 2023. It became generally available for Canadian customers in July this year, following launches in Australia, the UK and Frances.
Perrin set out to test whether the Canadian release “lives up to the hype” but concluded: “After several rounds of testing, I found Lexis+ AI disappointing. I encountered non-existent legislation referenced in the results (without a hyperlink), headnotes reproduced verbatim and presented as ‘case summaries,’ and responses with significant legal inaccuracies.”
Perrin’s first prompt was to ask Lexis + AI to draft a motion for leave to intervene in a constitutional challenge to a drug possession offence. He says: “Its response referenced ‘Section 15.07 of the Canada Legislation,” which does not exist. When I pointed this out, Lexis+ AI failed to acknowledge the error and displayed an automated message instead. This was a disappointing start, especially given the platform’s promise of reliable, citable results.”
He added: “However, I did notice that when Lexis+ AI provides a hyperlink to a case or statute, it’s not a hallucination. There was no such hyperlink for ‘the Canadian Legislation,’ so I learned to be aware of the lack of a hyperlink. What about the quality of the ‘draft motion’? Unfortunately, what was generated didn’t even qualify as a rough first draft.”
In his second interaction, Perrin asked Lexis+ AI to summarise the Supreme Court of Canada’s Reference re Senate Reform. “Instead of generating an original summary, it simply copied verbatim the headnote from the case (including the Supreme Court Reports page numbers), offering no added value,” Perrin said. “Even worse, when I requested a shorter summary, bizarrely, Lexis+ AI provided another verbatim summary, but this time of an entirely unrelated case involving a construction dispute from Alberta.”
One other prompt Perrin gave was ‘What is the test for causation in criminal law?’ which, Perrin says, provided an answer that confused criminal and tort law and got the test wrong.
Perrin’s initial tests of Lexis+ AI were run in August and September 2024, with an update re-running the same prompts right before submitting his review in late October 2024. In the re-run, Perrin says that when he asked for a summary of Reference re Senate Reform, Lexis+ AI once again reproduced the headnote verbatim. “When I followed up and requested ‘a shorter summary,’ it produced the verbatim headnote from the Supreme Court’s decision in Charkaoui v. Canada (Citizenship and Immigration),” he says.
All of the screenshots of his prompts are available to download here.
Perrin concludes that given Lexis+ AI’s “current limitations,” he cannot recommend it to his law students, and would not use it for his own legal research at this time.
LexisNexis has now responded in some depth to this article and in a statement, Jeff Pfeifer, chief product officer for LexisNexis North America & UK, told Legal IT Insider that the article by Perrin “does not reflect the response from customers to the product.”
Pfeifer said that LexisNexis has a customer-led development process and “we regularly receive feedback from users like, ‘The breadth of services offered is nearly staggering. I cannot think of a more comprehensive legal tool,’ and ‘I like the ease of use and the certainty that the tool is accessing already verified information in the Lexis system of products.’”
Interestingly, one head of KM who replied to Legal IT Insider today (18 November) with feedback on Lexis+ AI, albeit from a large UK firm, said: “We’re continuing to really reinforce the fact that any AI output needs to be checked and the user remains the author of the work, but the results are pretty positive and it’s going down well.”
As to whether Lexis+ AI should be rolled out to students, Pfeifer pointed out in his response to us that Canadian lawyers and law faculties have contributed to the development of Lexis+ AI and continue to provide important feedback. He said: “Since Canadian law faculty access started in July, we have sought and received valuable advice and guidance from law schools across the country. This semester, limited student trials are already in progress at a dozen Canadian law schools. This input has informed our approach to law school rollout and continues to improve the performance of Lexis+ AI.
“Nearly all of the law faculty, deans, and students we have consulted are supportive of providing Canadian law students with access to Lexis+ AI starting in January 2025. Canadian law firms are already using Lexis+ AI in their practices and LexisNexis is committed to preparing law students with the tools and skills they need as lawyers.”
Notably, Lexis hasn’t yet addressed the specifics of Perrin’s prompt failures, but Pfeifer does say that Lexis+ AI provides linked sources to mitigate against hallucination, observing: “The product recommends further review if a citation is not hyperlinked.” LexisNexis and Thomson Reuters Westlaw this year came in for criticism by Stanford University for suffering a significant rate of hallucinations, and Lexis emphasised then that it is important to understand that they were not promising perfection, but that all linked legal citations are hallucination free.
Pfeifer emphasised that Lexis grounds responses by using an enhanced Retrieval Augmented Generation platform, which now includes AgenticRAG, as well as incorporating techniques including proprietary metadata, the use of knowledge graph and citation network technology, and taxonomy services.
Perrin’s article is worth exploring in more depth, and we will be working with our lead analyst Neil Cameron, author of Legal IT Insider’s Generative AI Report. As an initial observation, Cameron notes that by section 15.07, Lexis+ AI might have meant section 15 of the Canadian Charter of Rights and Freedoms (on Equality), but he also notes that LexisNexis need to address the specifics of what went wrong with Perrin’s tests.
It is worth noting that in our Gen AI Report, Cameron ran comparative tests of legal specific solutions including (but not limited to) Lexis+ AI against ChatGPT, which in the case of the Reference re Senate Reform, gives a fairly in-depth answer. As for the causation point, ChatGPT also gets it, saying: “In Canadian criminal law, the test for causation requires that the Crown prove both factual and legal causation to establish the accused’s responsibility for the consequences of their actions.”
The Perrin Paper will undoubtedly fuel demands for transparent benchmarking of the output of generative AI tools, and Legal IT Insider is involved in one such initiative alongside Artificial Lawyer, led by CMS chief innovation and knowledge officer John Craske.
Pfeifer said: “LexisNexis has not been contacted by Professor Perrin, but we welcome the opportunity to explore his suggestions to improve the product experience.” More to come on this. Given the fact that people have said on LinkedIn that they can’t find a response from LexisNexis anywhere, you can find Pfeifer’s response in full below.
An opinion article by Professor Benjamin Perrin published on Nov. 12 in the Canadian Bar Association’s National Magazine makes an assertion about the capabilities of Lexis+ AI that does not reflect the response from customers to the product.
As part of our customer-led product development approach, Lexis+ AI customers tell us that the research answer quality, document drafting skills, and summarization capabilities in Lexis+ AI are significantly better than prior methods, saving them about 11 hours per week. We regularly receive feedback from users like, “The breadth of services offered is nearly staggering. I cannot think of a more comprehensive legal tool,” and “I like the ease of use and the certainty that the tool is accessing already verified information in the Lexis system of products.”
It’s important to note that Canadian lawyers and law faculty have contributed to the development of Lexis+ AI and continue to provide important feedback. Since Canadian law faculty access started in July, we have sought and received valuable advice and guidance from law schools across the country. This semester, limited student trials are already in progress at a dozen Canadian law schools. This input has informed our approach to law school rollout and continues to improve the performance of Lexis+ AI. Nearly all of the law faculty, deans, and students we have consulted are supportive of providing Canadian law students with access to Lexis+ AI starting in January 2025. Canadian law firms are already using Lexis+ AI in their practices and LexisNexis is committed to preparing law students with the tools and skills they need as lawyers.
At LexisNexis, our technology combined with our content, in addition to extensive system measures that include answer quality reviews by JD-trained attorneys, and our customer-led approach to developing products make Lexis+ AI a solution that is highly trusted, accurate, safe, secure, and can adequately handle nuanced legal concepts. Here are a few core examples of how we ensure this:
First, Lexis+ AI provides linked validation of citing authority references to underlying cases and legislation to help substantiate outputs and mitigate hallucination risks. The product recommends further review if a citation is not hyperlinked.
Second, in the fast-paced large language model (LLM) evolution, Retrieval Augmented Generation (RAG) and fine-tuning are the best methods available to ensure the highest answer quality. LexisNexis focuses on AI answer quality through an enhanced proprietary RAG platform, which now includes AgenticRAG capabilities for complex and nuanced legal queries. Our proprietary RAG infrastructure allows us to ensure that Lexis+ AI responses are grounded in our extensive repository of current, exclusive legal content, ultimately achieving the highest-quality answer with the most up-to-date validated citation references. We are seeing week-over-week improvements in answer and citation quality, thanks to ongoing technology development.
Third, LexisNexis has incorporated other techniques to improve the quality of our responses—proprietary metadata, the use of knowledge graph and citation network technology, and taxonomy services, to name a few. Each aids in the identification of the most relevant legal authority to support user questions.
Importantly, LexisNexis is responsibly developing legal AI solutions with human oversight. LexisNexis, part of RELX, follows the RELX Responsible AI Principles, considering the real-world impact of our solutions on people and taking action to prevent the creation or reinforcement of unfair bias. It is important that humans have ownership and accountability over the development, use, and outcomes of AI systems. We have an appropriate level of human oversight throughout the lifecycle of our solutions. This is core to ensuring the quality and appropriate performance of our solutions.
Professor Perrin suggested that users prefer AI-summarized cases to human-summarized cases. We appreciate his suggestion and will consider this in future product development. Additionally, we tested use cases not supported by Lexis+ AI. We clearly communicate drafting use cases, supported by the product and guide users to offer feedback about future cases they would like to see supported.
Ultimately, our goal is to provide the highest quality answers to our customers, including links to validated, high-quality content sources.
LexisNexis has not been contacted by Professor Perrin, but we welcome the opportunity to explore his suggestions to improve the product experience.