In what sounds like a budding lawyer’s stress dream, a new research paper picks apart OpenAI’s claim that GPT-4 managed to pass the bar exam with flying colors.

The paper, called “Re-evaluating GPT-4’s bar exam performance” and published this week in Artificial Intelligence and Law, questions both the large language model’s (LLM) score and its interpretation, as published in a pair of reports last year. It comes as the legal field has forged ahead with turning LLMs into research tools, but AI’s use in the legal profession has also led to some high-profile mistakes.

What the study says: OpenAI’s original attention-grabbing claim about GPT-4’s legal prowess traces to a technical report published by the company last year, which boasted that the LLM placed “around the top 10% of test-takers” on a “simulated bar exam.” A separate paper published before OpenAI’s report found that GPT-4 scored a 297 on the Uniform Bar Exam, high enough to pass in all jurisdictions.

But the Artificial Intelligence and Law research, authored by Eric Martínez, a PhD student at MIT and a graduate of Harvard Law School, argues that the first above claim is compared to a February batch of Illinois State Bar test-takers that “appears heavily skewed toward repeat test-takers who failed the July administration and score significantly lower than the general test-taking population.”

Keep reading here.—PK

What’s this? A streaming service that has:

• All the best, tech, engineering, science, travel, (and much more) video creators?
• Nile Red, Real Engineering, Wendover Productions, Downie Live? With no ads nor ad reads?
• Exclusive and highly produced, interesting documentaries, series, and even films that you can’t get anywhere else?
• Early access to videos from dozens of your favorite channels?
• Morning Brew Daily & Good Work with Dan Toomey now available ad free on the brand new curated News section?

Could it really exist? Yes it could. In fact, it does.

Subscribe to Nebula for less than $3 / month and get all of the above and much, much more. All while supporting creators making the best, entertainingly informative videos out there, now on the world's best independent video streaming platform.

The notion of selling pickaxes during a gold rush is a business cliché, but what about selling slide decks about gold-mining best practices?

That’s part of how Accenture has landed $600 million in new bookings from the generative AI craze in the first few months of this year, despite a diminished forecast for the consultancy’s year ahead overall, according to its recent quarterly earnings report. One of the executives at the center of this surge is Lan Guan, who was elevated to the position of Accenture’s chief AI officer last year after previously leading the company’s data and AI practice.

Accenture is far from the only company to umbrella booming AI operations under this trendy new C-suite title. In a survey published in CIO magazine last fall, 11% of midsize to large companies said they had hired a CAIO, and another 21% were seeking one.

But what do these AI-devoted execs actually do? And is this a position every company needs to fill? This is the second in a series of CAIO profiles in which Tech Brew looks into these questions.

In Accenture’s case, Guan said the creation of her perch was an acknowledgement of how much the company could stand to gain from client companies’ appetite for generative AI. Around the same time that she was promoted, the consultancy pledged to invest $3 billion in AI over the next three years and double its AI-trained workforce to 80,000 through hiring, acquisitions, and training.

“We definitely saw an enormous amount of opportunities in front of us to use generative AI together with classical AI to drive a lot of enterprise reinvention, especially post-pandemic,” Guan told Tech Brew. “So I think, simply put, that was the key, direct catalyst to create this role.”

Keep reading here.—PK

Coworking is a weekly segment where we spotlight Tech Brew readers who work with emerging technologies. Click here if you’d like a chance to be featured.

How would you describe your job to someone who doesn’t work in tech?

As chief product officer, I lead the definition of Hinge Health’s product vision and strategy, software and hardware development, user experience design, and consumer marketing. My job is to work with a talented group of inventors, designers, clinicians, and technologists to build a product that puts a physical therapist in your pocket. Basically, we help people experiencing muscle or joint pain by connecting members with physical therapists and health coaches who help design a personalized program that is accessible from a phone.

What’s the most compelling tech project you’ve worked on, and why?

Beyond my work at Hinge Health, one of the most exciting projects I ever worked on was the invention, development, and mass-market success of Chromecast at Google. In addition to the product being innovative in terms of technology, simplicity, and user value, it was also the seed of Google’s consumer hardware business.

Keep reading here.