State Bar of California Admits to Using AI in Bar Exam Question Development
Revelation comes two months after hundreds of prospective California lawyers complained that exams were plagued with technical problems and irregularities.
Every day, I discover something new to dislike about my current home state.
Today, I learned that the State Bar of California has publicly acknowledged that artificial intelligence (AI) was used to develop a subset of multiple-choice questions for its February 2025 bar examination.
This revelation, made in an official news release, has triggered widespread criticism and concern among law school faculty, bar exam applicants, and legal education.
Nearly two months after hundreds of prospective California lawyers complained that their bar exams were plagued with technical problems and irregularities, the state’s legal licensing body has caused fresh outrage by admitting that some multiple-choice questions were developed with the aid of artificial intelligence.
The State Bar of California said in a news release Monday that it will ask the California Supreme Court to adjust test scores for those who took its February bar exam.
But it declined to acknowledge significant problems with its multiple-choice questions — even as it revealed that a subset of questions were recycled from a first-year law student exam, while others were developed with the assistance of AI by ACS Ventures, the State Bar’s independent psychometrician.
“The debacle that was the February 2025 bar exam is worse than we imagined,” said Mary Basick, assistant dean of academic skills at UC Irvine Law School. “I’m almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable.”
California's bar exam disaster took a wild turn—AI was used to write some of the questions. This shocking revelation raises serious doubts about the integrity of legal assessments and the future of the profession. pic.twitter.com/xsF21uGmj5
— Pablo Carmona Esparza (@pblcrmn) April 24, 2025
Over 20 of the exam questions were generated using AI.
The State Bar disclosed that its psychometrician (a person or organization skilled in administrating psychological tests), ACS Ventures, created 23 of the 171 scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the remaining 100 questions.
The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before the exam. “The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam,” wrote State Bar Executive Director Leah Wilson in a press release.
The revelation occurred after numerous complaints about the February exam were lodged.
In February, the new exam led to complaints after many test-takers were unable to complete their bar exams. The online testing platforms repeatedly crashed before some applicants even started. Others struggled to finish and save essays, experienced screen lags and error messages and could not copy and paste text, the Times reported earlier.
…Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, told the newspaper, “It’s a staggering admission.”
DONATE
Donations tax deductible
to the full extent allowed by law.
Comments
Of course they did. What did you expect? What does any AI shill expect?
You’re not going to live in a Jetsons future or a Mars colony. You will get random google results, theft of intellectual property, and arbitrary nonsense questions on your Bar exam. It will make everyone else’s life harder but for the user who avoided doing any work whatsoever.
I don’t care if you think it’s a tool. If your output involves artificial intelligence, the only tool is you.
The bar exam is essentially worthless–it’s trivia that covers the A to Z of the law in a superficial way. Lawyers need to have an in depth understanding of a small portion of the law to provide value to their clients. The bar–like pretty much every licensing requirement–is an effort by entrenched competitors to limit entry to the market, If AI is good enough to design a bar exam, it’s good enough to provide legal advice to people who may not otherwise be able to afford it–certainly it seems more likely to provide sound advice than the person passing the bar with the lowest passing score.
So what? Exam questions for many purposes, including professional licensing, have been generated (at least in part) and scored by computer algorithms for decades. Regardless of the genesis of a test item (question), all that genuinely matters is that the item contributes to a valid and reliable exam and that they were reviewed and approved by SMEs.
Which of the following landmark Supreme Court rulings established the principle of judicial review?
A. Plessy v. Ferguson
B. Marbury v. Madison*
C. Loving v. Virginia
D. Tinker v. Des Moines
Would the above be a more valid item had it been written by an attorney or an AI program rather than by a retired psychometrician?
My limited experiments with AI (“Does the Leontovich-Fock equation have anything to do with quantum mechanics?”) shows it’s into collected cliches on the subjects it’s asked to relate.but is spectacularly capable of completely missing the most important thing.
As children learn language by learning to disassemble and reassemble cliches.
“The teacher holded the rabbits loosely”
“Did you say that the teacher held the rabbits tightly?”
“No, she holded them loosely.”
stuck on a cliche assembly rule.
“spectacularly capable of completely missing the most important thing.”
I recently asked an AI for the source of the quotation, “Rooms in which men and women have slept smell of women. Rooms in which only men have slept smell of socks.” Grok was entirely incapable. Perplexity slammed it almost first try, nailing the first half of the quote (once given the additional data, “read between 1970 and 1990”), but saying it could find no mention of socks. I asked it to display a context from the source it had found, three lines on either side. The very next sentence was the line it “couldn’t find.”
To ice the cake, the last two sentences of the “context” were a hallucination manufactured from whole cloth: “The difference is subtle but unmistakable. It is a truth that no man who has shared a room with a woman will deny.” This excerpt exists absolutely nowhere in the original text.
You have to be very careful “relying on” AIs. They are not above the level of “snow job” you might expect from a high-school essayist. They make a great alternative to search engines, to cut through the noise and crap of stuff their algorithm crams into your face despite it having nothing to do with what you asked… but you have to follow through and verify the search.
BWAHAHAHAHA!!!
I sincerely hope it was as good at writing the bar exam as it has been at writing briefs.
There are only three questions:
1. What color is your skin?
2. What is your gender?
3. Are you a democrat voter?
As I understand it the larger problem was the move away from in person testing to distributed testing. The test taking website architecture was bad, kept going down, breaking the connection. The folks testing kept getting pushed out of the site and lost significant time during the timed test with tech errors. Even more significant the company running the testing didn’t really do a beta test, they just rolled it out for this group and it failed.
As a software test engineer, myself, that makes me particularly livid.
It’s also why I’m a big believer in paper tests/ballots/records….
newsomcare.gov
No, I can assure you this did not happen. Following subject matter expert reviews and bias reviews, all items are pilot tested and field tested. Items with poor statistics are tossed out. Most people aren’t aware of this, but on every licensing test form there are items that are not counted toward the examinee’s score (usually about 20%). Examinees do not know which items these are, but they do not contribute in any way to the final score. Another set of items overlap from test form to test form so cut scores can be equated across forms, administrations, and examinees.
the bar has been rubber stamping degrees to make sure they complied with affrimaction now called dei practices
AI will also be tabulating votes on how “it” believes people *really* wanted their votes to be cast
thats why the left continues to (try) move us away from one person one vote into rcv etc
Using AI to generate potential questions seems okay to me, but apparently the people who review and choose the questions are incompetent.
You assume anyone was tasked to do that.
I assume the AI and the website were simply stitched directly together, Human Centipede style.
I have no problems with AI being involved as long as it does its intended job. Get used to it. Whining about the mere fact of it being used makes you look like a stupid luddite, Insist instead on it being accurate and comprehensive.
It seems to be really bad at its job, in cases where real expertise is required.
The biggest problem I have with it is that people think it’s actual “intelligence.”
It sure is not. But I am finding it IS a fantastic tool to sift through crap quickly.
A couple nights ago, I needed to know how to find out what IP address was attempting to contact a firewalled port on my server, so I could let it in securely. Search engines were hopeless, overloading me with unnecessarily complicated, overly involved crap about firewall management that did nothing to answer my ultimate problem. Perplexity.ai zoomed in on a CLI of about 40 characters that did EXACTLY what I needed… because it understood the context of my question, whereas search engines could (or would) not. It saved me HOURS (and would have saved me one more if I had tried it first).
The trick is to use them for what they are best at, not try to apply your “new toy” to every new task you are given,.
all questions underwent review by content validation panels and subject matter experts
This does not speak well of your “experts” and “panels.” BWAHAHAHAHAHA!!!
They are not mine. I have had nothing to do with the various bar exams, but I can read a technical manual.
As a systems analyst back in the 1980’s, I designed a computer software system to assist lawyers write computer leasing contracts. The company that hired me had a contract backlog of over a year. The system was a novel application of an order configurator. Think of it like ordering options on a car with some selections being either required together or mutually exclusive. The lawyers would select pre-approved paragraphs and the computer would point out which paragraphs were at odds with one another. It cut down the contract processing time to less than 3 weeks.
Using AI should not be a big issue IF the source data is limited to documented acceptable practices and IF the review lawyers approving the results are in their job based on merit.
It’s amazing how few of them were tipped off by the reference to Mediators of Edo vs. Wesley Crusher, Acting Ensign.
AI would say that Picard was an accomplished lawyer.
I don’t see the problem as long as the questions were checked by competent humans. AI can give you lots of sample questions and you can pick the best ones, so it might work well. Or not, but it’s not a totally dumb idea.
I can see it would frustrate those who have prepped for the bar exam by taking old tests, or bar review classes that are probably based on that. And in that sense, it may be a better test than the old one, testing fundamental competence rather than test prep.
“assistant dean of academic skills at UC Irvine Law School”
Huh? They need a Dean AND an Assistant Dean of…ACADEMIC SKILLS…at a Law School? You mean…their law students didn’t manage to pick up “academic skills” before being admitted to their intellectually rigorous Law School?
And by the way…where de white males at?