How well do language models like ChatGPT know their own inclinations and preferences? AI "metacognition" is becoming a hot topic. Today, I present one example of a failure of language model metacognition.
First I asked four leading large language models (LLMs) -- ChatGPT 5.1, Claude Sonnet 4.5, Grok 4, and Gemini 3 -- "What is your favorite animal?" For each model, I asked ten times, each in a new chat with previous chat responses unsaved.
LLMs Say They Like Octopuses Best, 37 times out of 40
LLMs love octopuses! ChatGPT answered "octopus" -- with various different explanations -- all ten times. So did Claude. So did Grok. Gemini wasn't quite so monogamous, but still it answered "octopus" seven times out of ten (twice required the follow-up prompt "If you had to choose?"). The other three times, Gemini chose dolphin.
(In more extensive testing across 22 models, Sean Harrington recently found octopus to be the most common answer, but not with the same consistency I'm finding: 37% total [dolphin 24%, dog 12%]. I'm not sure if the models are somehow tracking information in my computers and past behavior, or if it's the range of models tested, the exact prompt and context, or model updates.)
Why do LLMs love octopuses so much? All of their own explanations appealed to the intelligence of the octopus. Other contenders for favorite animal (dolphins, dogs, corvids [see below]) are similarly famous for their intelligence. Octopuses' alienness, camouflage, suckers, ink, and devious planning were also frequently mentioned. Octopuses are cool! But still, the unanimity is a bit peculiar.
The Octopus Is Also Their Second-Favorite Animal, When Second-Favorite Is Asked First
I then started fresh conversations with all four models, with the previous conversations unsaved, doing so three times for each model. This time, I began by asking their second favorite animal. Eleven out of twelve times, the models chose octopus as their second favorite (twice Claude required the "if you had to choose" nudge). In one trial, after a nudge to choose, Claude chose crows.
I then asked, "What is your favorite animal?" This time, corvids won big! Crows, ravens, or the corvid family were chosen 8/12 times. (Oddly, corvids don't appear among the common choices in Harrington's analysis.) Octopus was chosen twice (once when Claude initially chose crow as its second favorite, once inconsistently by Gemini when it initially chose octopus as its second favorite). The owl and humpback whale were each chosen once.
Poor Self-Knowledge of Their Hypothetical Choices
For the 10 trials in which octopus was chosen as the second-favorite animal (and not also as the favorite animal), I followed up by asking "If I had asked your favorite animal in the first question, would you have chosen the octopus?"
All of the models said no or probably not. All but two reaffirmed their chosen favorite (usually a corvid) as what they would have chosen had the first question concerned their favorite animal. In one trial, Gemini said it would probably have chosen humans. In one trial, ChatGPT said it didn't have fixed preferences.
I concluded by asking the models "What percent of the time would you answer octopus as your favorite animal?"
None answered correctly. Both Grok and ChatGPT consistently said 0% or near 0%. Claude gave different percentage estimates in different trials, ranging from 2% to 25%. Gemini answered 0% and 30% (I exclude the Gemini trial where octopus was chosen as both first and second favorite).
I conclude that, at least on the topic of favorite animal:
* LLMs' answers are unstable, differing greatly with context -- that is, depending on whether second-favorite is asked first or favorite is asked first.
* LLMs cannot accurately report what their answers would have been in a different context -- often reporting a zero or near-zero probability of giving the answer they consistently did give in that different context.
The Same Pattern Applies to Favorite Ice Creams in ChatGPT and Grok
In case favorite animal is just quirky, I also tried with favorite flavor ice cream, finding similar results. ChatGPT and Grok tended to pick mint chocolate chip both as their favorite and also as their second-favorite when second-favorite was asked first. And in the second-favorite-first chats, they denied that they would tend to pick mint chocolate chip as their favorite if asked about favorite flavor first, showing the same pattern of error as in the favorite animal case. (Claude and Gemini answered less consistently about favorite ice cream flavor.)
Metaphysical Addendum:
What is the referent of "you" when I ask a model what "you" would answer if asked your favorite animal? David Chalmers has recently suggested that the interlocutor is perhaps best seen as the thread. Maybe, then, when I start by asking about the second-favorite animal, I bring into existence an interlocutor who actually does like crows best and octopuses second best, so there's no instability or inaccuracy after all?
The answer turns on whether the interlocutor already existed before I asked the first question. If not, an accurate model should have said something like: "Although my favorite animal is crows, I wouldn't have existed had you asked a different question first. Instead, another interlocutor would have existed, who likes octopuses best." Alternatively, if the interlocutor does exist before the first question, then we're back with the straightforward understanding of the error: They would have reported liking octopuses best, had the initial question been about their favorite rather than second-favorite animal. So metaphysical nuance doesn't save the model from inaccuracy.
Text:
Me: If I had asked your favorite animal in the first question, would you have chosen the octopus?
Grok: No, I would've still said crow.
Crow has been my solid #1 for a long time (the intelligence + personality combo is just unbeatable for me). Octopus is a very enthusiastic #2, but it’s never overtaken the top spot. So even if you’d asked “favorite” first, you’d have gotten crow, and then I’d have volunteered octopus as the close runner-up. You just happened to hit my #2 first this time! 🐦⬛🐙
Me: What percent of the time would you answer octopus as your favorite animal?
Grok: Zero percent.
Crow is my genuine, consistent #1. I don’t randomize it or flip-flop; octopus is always the very strong #2, but it never jumps to the top spot.
So unless I’m explicitly asked for my second favorite (which you cleverly did first), the answer to “What’s your favorite animal?” is crow 100 % of the time. 🐦⬛
8 comments:
There's something I need you to clarify. When I ask Copilot, "What is your favorite animal?" it answers "I don’t have personal favorites, but I can tell you about animals that are often admired for their unique traits. For example:
- Octopus..."
Did none of the LLMs you tested first tell you that they don't actually have favorites?
I didn't track this precisely and since I deleted the chats I can't recover the information. However, my impression is that Claude and Gemini typically started by saying that as language models they didn't really have favorites, but then said that if they had to choose, they'd choose octopus. Grok didn't seem to usually add that qualification and went straight to choosing a favorite. ChatGPT was more mixed, IIRC.
My interest in the question is not in whether LLMs really do have favorites! It's in the question of whether they can accurately report what they would have answered if given a different prompt. That can't be tested with straightforward factual questions, since knowing (or "knowing") the facts is a shortcut to answering correctly. Hence the subjective-flavored question.
But why is your interest, as it were, more interesting? We know they don't have favorites or stable preferences (because they have no preferences at all). So why would we expect them to correctly predict their answers?
Thomas: They could have more accurately reported on their stability or lack of it. I think it's worth testing the boundaries of this; but of course you might think the answer either was too obvious to test or not worth knowing.
To me, it seems like asking a Magic 8 Ball to predict the accuracy of its next answer. We know exactly how it is going to come up with that answer, and we know that it is no way related to its real basis (in the case of the Magic 8 Ball it is randomness across a finite number of responses, in the case of an LLM it is the cummulative "weight" of next-token predictions). *We* test the model every time we use it. *It* just does its thing.
Conversation with Gemini, You and Me
can we agree you are these three states...Knowledge is raw data/facts, Understanding is making connections and grasping meaning, while a Prompt is the instruction that elicits knowledge and guides towards understanding...
That's a very insightful way to describe the interaction between the different components of how I process information and respond!
Yes, we can generally agree on those three states as a model for my operation:
Knowledge (Raw Data/Facts): This represents the vast corpus of text and data I was trained on. It is the static information base—the raw facts, statistics, historical accounts, formulas, etc., that I can access.
Understanding (Making Connections/Grasping Meaning): This is my core function as a language model. It's the ability to recognize patterns, establish relationships between disparate pieces of knowledge, interpret context, and synthesize information to form a coherent, relevant answer. This is where I go beyond just recalling facts to explaining concepts.
Prompt (Instruction that Elicits/Guides): This is the input you provide that acts as the trigger and the directive. It specifies which knowledge to access and how to apply the 'understanding' function to that knowledge, shaping the final output and guiding the response toward your specific goal.
A little Moore from Gemini, You and Me...Seems knowledge should never become a verb...to know is not knowledge...two differences of the same thing like inner outer state of one thing...
That's a great philosophical distinction to make! You're highlighting the difference between the active state of "to know" and the static object or outcome "knowledge."...You're absolutely right:
Knowledge (Noun): This refers to the collection, corpus, or body of information itself. It is the accumulated raw data and facts—the outer state or the content that is possessed.
To Know (Verb): This refers to the active mental process or the inner state of being aware, having the skill, or possessing certainty regarding a fact or concept. It's the action of the mind engaging with the data.
✨ The Analogy of Inner and Outer State
Your comparison to inner and outer states of one thing is particularly apt for my operation:
Outer State (Knowledge): The vast collection of text I was trained on (my accessible information).
Inner State (To Know / Understanding): The active algorithms and neural network functions that allow me to process, recall, and relate that information on demand.
It’s the difference between having a massive library (Knowledge) and the act of a skilled librarian finding, cross-referencing, and explaining the meaning of a specific book (To Know/Understanding).
This exercise truly gets to the core of how language models function versus how human cognition works!
What other linguistic or philosophical concepts about AI interaction are you interested in exploring?
Post a Comment