BasedBench: an evaluation benchmark for meme and visual joke understanding
How based is your model?
Take an image of a meme or visual joke that is non-trivial for humans
to understand. Ask a VLM what it means. That's it: that's the
tweet task.
Ground truth
Model responses
Why BasedBench?
Memes are fun. AI is fun. But also...
- We need more than another coding eval. LMs are taking over the world, their capabilities are spiky, and labs are primarily focused on a few high-value domains. But we deserve models that are less flat, more well-rounded, and more based.
- Meme understanding tests multiple skills at once: visual reasoning, image text recognition, deep-cut cultural references, recent knowledge of current events, and the ability to connect the dots across multiple references into a single explanation.
- The benchmark is "labeler free", inspired by SWE-Bench; it leverages found, real world data from "explain the joke" subreddits, where humans ask what a joke means and other humans explain it.
What's in the box?
- 519 human-verified memes with ground truth, model predictions, and LLM-as-a-judge scorers, available in the BasedBench dataset on Hugging Face.
- A data browser experience on Hugging Face Spaces.
- A leaderboard tested across five models: Gemini 3.1 Pro, GPT-5.5, Muse Spark 1.1, Claude Opus 4.8, and Grok 4.3.
- Source code is available on GitHub.
Shut up and show me the results memes!
Without further ado, here are the results:
Pass rate
519 validated examples, higher is better.
To great surprise, Gemini 3.1 Pro is the most based model among the competitors!
It comes in at 85% correct meme understanding with GPT-5.5 right on its heels at 81%. Meta's Muse Spark 1.1 enters the leaderboard in third at 76%. These three are far ahead of Claude Opus 4.8 which disappointed relative to my expectations. Given the context of xAI, which also runs the extremely based x dot com (the everything site), you might imagine it would punch above its weight, but alas, it is a weaker model here.
Takeaways and examples
Superhuman performance
The top-level takeaway: today's VLMs already excel at this task. I did not perform a rigorous human baseline, but I would estimate my own pass rate at 20%. One clear advantage for VLMs is their encyclopedic knowledge, which includes orders of magnitude more references: TV, movies, music, history, literature, all of it. That does indeed contribute to their superhuman performance. But they are also extremely capable in other areas, like "connecting the dots" between seemingly unrelated elements and correctly identifying which details are central to the humor. Here are a few examples where model performance impressed and surprised me.
Ground truth
Model responses
Subhuman performance
There are very few examples where all LLMs fail but the human baseline passes. Here they are:
Ground truth
Model responses
Dot connecting
Some jokes are not solved by naming one reference. The model has to assemble several small clues, decide which details matter, and turn the pieces into one coherent explanation.
Ground truth
Model responses
Visual recognition, reasoning, and understanding
Other misses start much earlier: the model has to see the image correctly. These examples turn on hidden objects, altered images, reading order, optical illusions, and tiny visual cues that change the whole joke.
Ground truth
Model responses
Fully solved
These 20 examples are sampled from the memes where all five models passed (204 total!). The fact that so many memes are already fully solved and saturated is, I think, quite incredible.
Ground truth
Model responses
Future directions
The most natural next step to expand on this work is to move from a static benchmark to a dynamic one that runs on a cron, regularly appending new memes to the eval set.
The biggest challenge to make that possible is improving the accuracy of the AI filters and judges. To arrive at the 519 instance human-verified sample, I reviewed ~2000 total memes and rejected 1400 that either did not express a meaningful joke, were trivially easy, or otherwise felt like they did not suit the spirit of the task. Nailing the pipeline and prompts for data filtering to ensure that all examples are meaningful and hard enough is a necessary step to unlock a useful dynamic eval set. One approach I plan to investigate is using an optimizer like GEPA rather than fully hand-crafting the filter and judge pipeline.
Another direction concerns confabulation and abstention. This benchmark makes no distinction between an abstaining response and one which confidently confabulates an incorrect answer. VLMs should know what they know and profess ignorance when they don't know. A future version might treat confabulation and abstention differently, taking away points for confabulation (or giving partial credit for appropriate abstention).
Finally, BasedBench does not ask models to judge whether something is a "good" (based) meme. In earlier versions the task asked VLMs to explain the mechanisms of humor behind the meme. This might have provided an inroad into the squishier question of meme quality, the idea being that if a model understands why something is funny, it has at least some understanding of the "magnitude" of that why, e.g. whether that mechanism is fulsomely realized or fails to come off. But it was not meant to be... Firstly because the models somewhat collapsed into highlighting "absurdity" and "contrast" in almost every case rather than giving more differentiated and specific mechanisms of humor. And secondly because on reflection I was not sure myself that I could give a substantive and true explanation of what made the based memes based and the cringe memes cringe. For now this remains consigned to the realm of vibes.