We tested our own legal AI and published the results: 160 of 386 files could not be cited
On 24 October 2026 we audited the entire LexArab legal library and tested the assistant on Moroccan, Jordanian and Iraqi law. 160 of 386 files turned out to be uncitable, and a country-matching bug was routing banking questions to Egyptian law. Here is what we found, what we fixed, and what we did not.
We tested our own legal AI and published the results
Why publish this
The easiest claim in legal AI is that your assistant is accurate. The hardest thing is to test it and publish what you find. On 24 October 2026 we audited the whole LexArab legal library and ran the assistant against known questions. These are the results.
The test
Three questions, each with one correct answer in the text of the law: the limitation period for tort under Moroccan law; the limitation period for a damages claim under the Jordanian Civil Code; and the penalty for simple theft under the Iraqi Penal Code.
Before
Morocco: the substance was right, but the assistant attributed three articles to a library file that did not contain them. It retracted this itself in the next message, which is to its credit, but the misattribution should never have happened.
Jordan: the answer was correct, yet all six sources it relied on were Omani and Iraqi. Not one Jordanian file appeared.
Iraq: the assistant declined to name the article. The sources shown were Libyan and Qatari laws, unrelated to the question.
What was actually wrong
Not the model: the library and the search.
Country names were matched as substrings rather than whole words. The Arabic word for "the bank" contains the Arabic word for "Egypt", so banking questions were being filtered to Egyptian law. The word for "extinguishing" contains the word for "Muscat", so a Jordanian limitation question read as both Jordanian and Omani, and the filter was dropped entirely.
More seriously, of 386 files in the library, 160 could not be cited from: around a hundred had sound Arabic text but had lost their article numbering during PDF extraction; around forty were English or French translations rather than Arabic; around twenty had 3 to 11 per cent of characters corrupted by font encoding.
In the Iraqi Penal Code file, the Arabic phrase for "simple theft" does not appear anywhere. The assistant's refusal was not incapacity but honesty: the text was not in front of it.
What we fixed
Whole-word matching for country names. Disambiguation of Oman and Amman. A quality gate at upload that now also detects non-Arabic files and files without article numbers, where before it only caught character corruption. A re-scan of all 386 files, with the failures flagged to the administrator. And the assistant now asks which country you mean rather than answering from another country's law.
After
Jordan: the assistant now quotes Article 272 of the Jordanian Civil Code verbatim, with Jordanian sources only, where before there were none. Iraq: all sources are now Iraqi. Morocco: the misattribution is gone, and the assistant says plainly when the article text is not in the file.
What is still broken
The Iraqi question still has no specific answer, because the file is a poor scan. Only replacing it will fix that. This is the honest limit of the technology: no model can cite a text it does not have.
The lesson
The quality of a legal assistant lies less in the model than in the texts it holds. The real test is not whether an answer looks right, but whether the sources shown come from the right country and actually contain the quoted text.
If you are evaluating any Arabic legal AI, ask a question whose answer you know in a specific jurisdiction, then inspect the sources: which country are they from, and does the quoted provision appear in them? That single check tells you more than any marketing page.
This article is for information only and is not legal advice.