I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.

Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.

His conclusion:

> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.

> However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest

Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this.

Citation: https://www.rei.com/c/bikes

Edited to add:

As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works.

1. https://simonwillison.net/2025/Nov/13/training-for-pelicans-...

Take a look at the GLM 5.2 and Deepseek V4 "animal on a plane" examples. In every case, the animal is standing on top of the plane, a clear misunderstanding of the concept of "animal on a plane"... with the exception of the Otter. The otters are sitting in a seat on the plane, looking out the window.

That's Ethan Mollick's "Otter On A Plane Using WiFi" image benchmark.

https://www.oneusefulthing.org/p/the-recent-history-of-ai-in...

(Sometimes the Racoon is sitting inside the plane as well, but the racoon is a common backup benchmark. I'm surprised it wasn't also holding a sign saying that it loves trash.)

Also, Grok seemed to really really enjoy "whale on a plane" in that second round, and kudos to GPT Terra for deciding after 3 rounds that the user was terrible at spelling and generated "Antelope On A Plain".

EDIT: I promise I'm a human, but I did just notice my "that's not x... that's y" construction at the start. I am rather Claudepilled — my apologies.

Me: how many P's are in the following text? [Pasted text]

Claude: There are 14 P's, all lowercase (no capital P's)

Me: how many in "strawberry"?

Claude: there are 3 R's in the word "strawberry".

Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified.

My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test

Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

However, I'm very surprised that most of the models make the same sideways mistake with only some of the animals, and they do it consistently.

With most of the models, cat, raccoons, and otters are almost always riding sideways. Why is that?

Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image.

It seems that we're missing a kind of step to decompose an image into a list of instructions (say, SVG paths, or even brush strokes with a real brush) to reproduce it properly. Doing so would probably need a true understanding of the structure of the scene, which is something that AI still struggles with to this day.

https://x.com/CrimeDecoder/status/2080008114615537766

Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.

Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?

So they aren't pelicanmaxxing but they are benchmaxxing in a way. The benefit of the pelican was originally that uplift on the pelican signaled an overall uplift on intelligence. I don't believe that is the case anymore and it is just another jagged edge of model intelligence.

Your next iteration will need different animals and different transportation options. You'll run out after a few iterations.

I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.

Tomato, tomato

aren't some LLM going to digest that thread at some point and indirectly learn from it?

basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content, some models are going to get a benefit in education?

you'd need the "AI" equivalent of an old-school "google whack", something with no previous results

* https://en.wikipedia.org/wiki/Googlewhack

Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.

Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.

I think we should stop using pelican benchmark.

Test the LLLM against things you want it to do.

Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.

Remember these Microsoft interview questions designed to identify the best developers?

"If you could eliminate one U.S. state, which one would it be?"

"How would you move Mount Fuji?"

Absurd interview questions have an air of legitimacy due to the quasi sophisticated justifications put forward for why they are good tests.

Absurd interview questions are not good tests of people or LLMs.

Relevant questions are good tests.

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Or the more pop layman version

"When a measure becomes a metric/KPI, it ceases to be a good measure."

Story time, I live in Argentina, and we don't have Big Macs, the main Mc Donald's brand, here, because during the CFK presidency, one of her tactics was to Goodhart economic metrics. Even the informal obscure ones like the [Big Mac Index](https://en.wikipedia.org/wiki/Big_Mac_Index), I don't know the precise details, but the Big Mac ended up being a very cheap item, like 2 or 3 times cheaper than actual menu items, but it was never on the advertised menu, and it also ended up being very small compared to the other burgers, so it wasn't even like a hack, a shrinkflation type of deal.

But hey, anyone who read the Big Mac Index table would never find Argentina at the bottom of that list along with a couple of other countries with bad brands, so the ploy worked. And now we live with the aftershock, the brand never really turned around, other brands with ridiculous names took over it like the McTasty, which makes me sound like that skit from Tarantino's Pulp Fiction.