In Latest Test of Its Prowess, ChatGPT Bot Passes Law School Exam
Conveniently, to allay cheating concerns, ChatGPT maker has also released AI detection tool.
I have recently begun following the developments related to ChatGPT (Chat Generative Pre-trained Transformer), a chatbot launched by OpenAI this past November. The New York Times hailed it as “the best artificial intelligence chatbot ever released to the general public.”
I also noted that the chatbot passed a Wharton Business School test as well as the US Medical Licensing exam . . . it is that good.
In the latest test of its prowess, it has just passed a law school exam….at the bottom of the class, though.
A chatbot powered by reams of data from the internet has passed exams at a U.S. law school after writing essays on topics ranging from constitutional law to taxation and torts.
….Jonathan Choi, a professor at Minnesota University Law School, gave ChatGPT the same test faced by students, consisting of 95 multiple-choice questions and 12 essay questions.
In a white paper titled “ChatGPT goes to law school” published on Monday, he and his coauthors reported that the bot scored a C+ overall.
While this was enough for a pass, the bot was near the bottom of the class in most subjects and “bombed” at multiple-choice questions involving mathematics.
“In writing essays, ChatGPT displayed a strong grasp of basic legal rules and had consistently solid organization and composition,” the authors wrote.
But the bot “often struggled to spot issues when given an open-ended prompt, a core skill on law school exams.”
The Department of Deadwood is now accepting applicants at all levels.https://t.co/Qrym40Mm3q
— wretchardthecat (@wretchardthecat) January 31, 2023
In light of such results, educators are becoming increasingly concerned about cheating. But with challenges come opportunities.
The maker of ChatGPT is trying to curb its reputation as a freewheeling cheating machine with a new tool that can help teachers detect if a student or artificial intelligence wrote that homework.
The new AI Text Classifier launched Tuesday by OpenAI follows a weeks-long discussion at schools and colleges over fears that ChatGPT’s ability to write just about anything on command could fuel academic dishonesty and hinder learning.
OpenAI cautions that its new tool – like others already available – is not foolproof. The method for detecting AI-written text “is imperfect and it will be wrong sometimes,” said Jan Leike, head of OpenAI’s alignment team tasked to make its systems safer.
“Because of that, it shouldn’t be solely relied upon when making decisions,” Leike said.
There have been some discussions about the types of jobs that ChatGPT and other advanced chatbots could replace. There is some good news to be had there. In addition to progressive social media writers, human resources and press offices could also be impacted.
David Autor, an MIT economist who specializes in labor, pointed to some mid-level white-collar jobs as functions that can be handled by AI, including work like writing human resources letters, producing advertising copy and drafting press releases.
“Bots will be much more in the realm of people who do a mixture of intuitive and mundane tasks like writing basic advertising copy, first drafts of legal documents. Those are expert skills, and there is no question that software will make them cheaper and therefore devalue human labor,” Autor said.
In conclusion: The excellent team of lawyers and contributors here is unlikely to be replaced soon!
DONATE
Donations tax deductible
to the full extent allowed by law.
Comments
In the latest test of its prowess, it has just passed a law school exam….at the bottom of the class, though.
Sounds like a certain former senator from Delaware.
This means it’s at least as smart as Biden was BEFORE the dementia.
Which is a rather low bar.
But the big one that it had to hop to run for president.
Someday the president’s pronouns MAY actually be beep/bop/boop.
I’m really looking forward to the future when all of our work is replaced by robots and we are free-spirited, rich, and infinitely happy.
That’s how this works, right?
A just machine to make big decisions /
Programmed by fellows with compassion and vision /
We’ll be clean when that work is done /
We’ll be eternally free and eternally young /
oooooooh…. /
What a beautiful world that will be /
What a glorious time to be free…
–STEELY DAN
Anna Nicole Smith thought that.
Just how much computing power and space is needed to run this thing?
A duct-taped array of iPhones just slightly smaller than a breadbox.
I’ve poked around with ChatGPT, and for my purposes, I didn’t find it useful. It is mundane, so passing something like a law school exam isn’t surprising, and I don’t know why it’s news. You could make a program that just scours google and do that. If you ask it to write a paper on something, it’s as basic as it gets. If I were in school I wouldn’t use that to turn in any paper. I might use it as a skeleton but the need to rework it a lot would be necessary, which is why you see it not being crazy successful at such things at this law school.
Tell me how different it is at this stage (key word) than copying from an encyclopedia back in the stone age. I’m not saying there aren’t concerns, because the encyclopedia couldn’t adapt like AI. But at this point (key word), I don’t see the big deal. And it’s not like we can put this genie back in the bottle. I have no idea what things will be like in a generation or two or far beyond. Imagine having AI-supervised government control. It would not shock me if Joe Biden right this second was really an animatron that will be mailed to Disney’s hall of presidents in January 2025.
There is an AI art generator called Midjourney. and it has limitations. But the creative ability of Midjourney is much better than that of ChatGPT. In fact, someone won a state fair art show blue ribbon last year, I think Colorado, without telling them it was generated with Midjourney. I’m not good enough to win an art show but I’ve been able to produce some pretty cool stuff.
Anyway, interesting times….
TY Colonel, I have been wondering about the art generators..
BTW, my first question to ChatGPT was about Elon Musk becoming President of the USA… at first it said yes, he could. I never saved the conversation.. but then it contradicted itself. It did pretty much the same thing one week later. It also told me that it was fed information up to 2021. I was not impressed.
What’s unique about it is that it doesn’t plagiarize.
It’s “smart” enough to write its own composition from the source material and cite it appropriately, given the right prompts.
It is most disquieting for the already-declining standard of effort being put forth by current academic pursuits.
I know many think these are worthless, but some elements like quantitative reasoning, research and evaluation of data, innovative intellectualism, and logical, qualitative reasoning are invaluable for the human individual. These can’t be appropriately replicated and outsourced to AI for anything high stakes in the real world. Academic pursuits are supposed to be a practice round for applications with higher stakes in the future. If the practice rounds are outsourced by the lazy doing a poor but passable job so they can YOLO no FOMO, then these same a-holes will slip into industries and infect them with rot. Your future health care as you age, our national defense/security, and many apolitical scientific pursuits will be among the hardest hit.
“What’s unique about it is that it doesn’t plagiarize.
It’s “smart” enough to write its own composition from the source material and cite it appropriately, given the right prompts.”
So then, even WAY smarter than Biden before the dementia.
That’s why I tossed in some qualifiers. It can pass a law exam? Big deal. And if you ask it the same question to write something, or with minimal variations, it doesn’t come up with as much randomness at all. At this stage, it is really not that smart or original. I’ve spent hours with it and after a couple days I never went back to it. But the AI art thing? The visual randomness and possibilities are far more superior than the verbal. I’ve toyed around with that a lot.
Regardless, this stage of AI is not what scares me.
Literary critics are safe.
“ChatGPT maker has also released AI detection tool.”
It analyzes the pattern of fnords in the generated text, same way you can analyze the pattern of yellow dots a laser printer secretly deposits on the page.
Beginning of the end. At least for liberal arts.
My goodness! Does that mean it is racist?
I thought about this: passing a bar exam SHOULD be easy for an computer taking an open book test. After all, the bar exam’s biggest challenge is memorization, issue spotting, etc in a timed environment. A computer has no memory challengs, nor time constraints. There are a limited number of issues to be spotted in any bar exam – a computer can juggle them all at the same time and a well software can easily assemble a response to a question in a very short time, It’s kinda like putting an finite number of pieces of a puzzle in place, but processed at the speed of light.
What I found that Chat AI cannot do is cite and analyze cases – something a bar exam never requires.
The fact that Chat AI ONLY got a C+ is surprising.
If there are “a limited number of issues” then someone will have the idea to list them and train the AI on those issues. So far apparently that has not happened.
Math is a surprising weakness of ChatGPT at this time. It could fix that completely by dumping all its math calculations to Mathematica. This obviously has not been done either.
And a combination with the art-generating AI is an intriguing thought.
We shouldn’t evaluate these AI’s on their own, but see how they can work in concert. They work in concert to help us, because if I have a math problem I’ll choose to give it to Mathematica, not ChatGPT. They could also work in concert against us.
I agree with TheFineReport here, especially in the context of the MultiState Bar Exam. That exam is a series of multiple choice questions that generally involve the common law. The common law is an extremely logical set of rules. Once you get the logic, it shouldn’t be too hard for a computer to figure out whether the answer is A, B, C or D.
As best I can remember from July 1984, a problem with the MultiState Exam is that some questions have no really good answers. They have wrong answers and one that is OK but not a really good answer.
Now they need to come up with a minority ChatGPT that will flunk the test and demand redress.
Next up: an AI auto-grader that figures out which exam came from a minority candidate, and gives those exams A grades.
ChatBot bombed on multiple choice […]
Properly designed multiple choice defeat chatbot. So, design an AI to quiz chatbot (this may not work since Winograd Challenge tests confound AI tech..)