When an organization releases a brand new AI video generator, it’s not lengthy earlier than somebody makes use of it to make a video of actor Will Smith consuming spaghetti.
It’s grow to be one thing of a meme in addition to a benchmark: Seeing whether or not a brand new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself parodied the pattern in an Instagram publish in February.
Google Veo 2 has achieved it.
We are actually consuming spaghett eventually. pic.twitter.com/AZO81w8JC0
— Jerrod Lew (@jerrod_lew) December 17, 2024
Will Smith and pasta is however one among a number of bizarre “unofficial” benchmarks to take the AI neighborhood by storm in 2024. A 16-year-old developer constructed an app that provides AI management over Minecraft and exams its means to design constructions. Elsewhere, a British programmer created a platform the place AI performs video games like Pictionary and Join 4 towards one another.
It’s not like there aren’t extra tutorial exams of an AI’s efficiency. So why did the weirder ones blow up?
For one, most of the industry-standard AI benchmarks don’t inform the common particular person very a lot. Corporations usually cite their AI’s means to reply questions on Math Olympiad exams, or work out believable options to PhD-level issues. But most individuals — yours actually included — use chatbots for issues like responding to emails and basic research.
Crowdsourced {industry} measures aren’t essentially higher or extra informative.
Take, for instance, Chatbot Arena, a public benchmark many AI fanatics and builders observe obsessively. Chatbot Enviornment lets anybody on the net fee how nicely AI performs on specific duties, like creating an online app or producing a picture. However raters have a tendency to not be consultant — most come from AI and tech {industry} circles — and solid their votes based mostly on private, hard-to-pin-down preferences.
Ethan Mollick, a professor of administration at Wharton, not too long ago identified in a post on X one other downside with many AI {industry} benchmarks: they don’t examine a system’s efficiency to that of the common particular person.
“The truth that there should not 30 totally different benchmarks from totally different organizations in drugs, in regulation, in recommendation high quality, and so forth is an actual disgrace, as persons are utilizing methods for this stuff, regardless,” Mollick wrote.
Bizarre AI benchmarks like Join 4, Minecraft, and Will Smith consuming spaghetti are most actually not empirical — and even all that generalizable. Simply because an AI nails the Will Smith take a look at doesn’t imply it’ll generate, say, a burger nicely.
One professional I spoke to about AI benchmarks instructed that the AI neighborhood deal with the downstream impacts of AI as an alternative of its means in slender domains. That’s smart. However I’ve a sense that bizarre benchmarks aren’t going away anytime quickly. Not solely are they entertaining — who doesn’t like watching AI construct Minecraft castles? — however they’re simple to know. And as my colleague Max Zeff wrote about recently, the {industry} continues to grapple with distilling a know-how as advanced as AI into digestible advertising and marketing.
The one query in my thoughts is, which odd new benchmarks will go viral in 2025?
TechCrunch has an AI-focused e-newsletter! Sign up here to get it in your inbox each Wednesday.
