An influential California legislator is pressuring the State Bar of California to ditch its new multiple-choice questions after a February bar exam debacle and revert to the traditional test format in July.
“Given the catastrophe of the February bar, I think that going back to the methods that have been used for the last 50 years — until we can adequately test what new methods may be employed — is the appropriate way to go,” Sen. Thomas J. Umberg, chair of the state Senate Judiciary Committee, told The Times.
Thousands of test takers seeking to practice law in California typically take the two-day bar exam in July. Reverting to the national system by the National Conference of Bar Examiners, which California has used since 1972, would be a major retreat for the embattled State Bar. Its new exam was rolled out this year as a cost-cutting measure and “historic agreement” that would offer test takers the choice of remote testing.
Alex Chan, an attorney who chairs the Committee of Bar Examiners, which exercises oversight over the California bar exam, told The Times earlier this week it was unlikely that the State Bar would revert to the NCBE exams in July.
“We’re not going back to NCBE — at least in the near term,” Chan said.
The NCBE’s exam security would not allow any form of remote testing, Chan said, and the State Bar’s recent surveys showed almost half of California bar applicants want to keep the remote option.
Last year, the financially strapped State Bar made the decision to cut costs by replacing the test questions developed by the National Conference of Bar Examiners’ Multistate Bar Examination, which does not allow remote testing. If the State Bar developed its own questions, it figured it could save money by sparing the expense of renting massive exam halls for all test takers.
The State Bar hired a vendor, Meazure Learning, to administer the exam and announced an additional $8.25-million five-year deal authorizing test prep company Kaplan Exam Services to create multiple-choice, essays and performance test questions.
But after the botched rollout of the new exam in February — when many test takers complained of a litany of technical problems, glitches and irregularities — the state’s highest court, which oversees the State Bar, directed the agency to plan on administering the July exam in the traditional in-person format.
The Supreme Court has yet to direct the State Bar to return to the NCBE system, even though test takers complained that some of the multiple-choice questions in the new test included typos and questions with more than two correct answers and left out important facts.
This week, the State Bar enraged test takers and legal experts when it revealed that it had hired ACS Ventures, its independent state psychometrician that validates and scores exams to ensure they are reliable, to develop a small subset of multiple-choice questions using artificial intelligence.
“They have to go back to the multi-state bar exam this summer,” said Katie Moran, an associate professor at the University of San Francisco School of Law. “They have just shown that they cannot make a fair test.”
Erwin Chemerinsky, dean of the UC Berkeley Law School, agreed.
“The reality is that remote options, as shown in February, work poorly,” he said. “The bar exam is too important for them to experiment as they did and are continuing to do.”
State Sen. Umberg, a former prosecutor, likened having a non-lawyer using artificial intelligence to draft questions for a bar exam “to non-physicians designing questions with the help of AI to decide who’s qualified to be a surgeon.”
As chair of the Senate Judiciary Committee, Umberg wields considerable clout over the State Bar. He recently pushed Senate Bill 40, a new law that requires the state Senate to confirm future appointments of the State Bar’s executive director and general counsel. After the February exam debacle, Umberg filed legislation to launch an independent review of the exam by the California State Auditor, to find out what went so “spectacularly wrong.”
That bill is slated to be reviewed at a May 6 Senate Judiciary Committee hearing, along with Senate Bill 253, the State Bar’s annual license fee authorization bill, which gives lawmakers leverage to push the Bar to make improvements.
Umberg said the upcoming hearing will go beyond just the administration of the February bar exam.
“We’re going to be looking at the leadership of the bar,” he said. “We’re going to be looking at what’s happened since our last oversight hearing in terms of accountability and transparency.”
Asked whether he had confidence in State Bar leadership. Umberg said: “My confidence is shaken.”
Umberg would not say whether the State Bar’s executive director Leah Wilson should step down, but said the question is “one of the issues that we’ll be examining here in the months ahead.”
The State Bar announced this week it will ask the Supreme Court to adjust test scores for those who took its February bar exam.
For critics of the State Bar, the problem is not just that it used AI to develop questions, but that it did so without the knowledge of the California Supreme Court and the Committee of Bar Examiners.
The State Bar told The Times the decision to have ACS Ventures develop questions with the assistance of AI programs “was made by staff within the Admissions office and not clearly communicated to State Bar leadership.”
“This was a breakdown, and structural changes have been made within Admissions to address it,” the State Bar said, noting that it has since created a new chief-level role over Admissions reporting directly to the Executive Director and “a new team structure to strengthen accountability and effectiveness.”
At the same time, the State Bar downplayed the significance of hiring ACS Ventures to develop questions, noting the company’s “general support of the bar exam — of which the CBE and Board are aware — is covered by their existing contract.”
All multiple-choice questions, including those “developed initially with the assistance of AI,” the State Bar said, “were subsequently reviewed by content validation panels comprised of lawyers, and an attorney subject matter expert, as part of the question development and finalization process.”
Whatever happens next, Umberg said, the State Bar should take the time to prevent the debacle of the February bar exam from happening again.
“Taking the bar exam, it’s really a test that people prepare for for three or more years,” Umberg said, noting two of his children had taken it in recent years. “The fact that the test takers, in essence, were guinea pigs for the February bar is absolutely unacceptable.”
“That’s why,” Umberg said, “we’re going to go back to the old methodology here in July.”
