You've probably heard of GPT-3, the hot new language model that can produce strikingly humanlike language outputs in response to ordinary questions – basically, the world's best chatbot. (Google's LaMDA, a similar type of program, has also recently been in the news.)
With Daniel Dennett's cooperation, Anna Strasser, Matthew Crosby, and I have "fine-tuned" GPT-3 on millions of words of Daniel Dennett's philosophical writings, with the thought that this might lead GPT-3 to output prose that is somewhat like Dennett's own prose.
We're curious how well philosophical blog readers and people with PhDs in philosophy can distinguish Dennett's actual writing from the outputs of this fine-tuned version of GPT-3. So we've asked Dennett ten philosophical questions and recorded his answers. We posed the same questions to GPT-3, four times for each of the ten questions, to get four different answers for each question.
We'd love it if you can take a quiz to see if you can pick out Dennett's actual answer to each question. Can GPT-3 produce Dennett-style answers sufficiently realistic to sometimes fool blog readers and professional philosophers?
UPDATE, July 15: We have collected enough responses to begin analysis. Please feel free to take the test for informational purposes. We will be able to see your responses, but we will not check regularly nor automatically report your score. If you take the test and want your score, email me at my academic email address.
This is a research study being conducted on the internet platform Qualtrics. It will take approximately 20 minutes to complete. Anyone is welcome to participate.
If you're interested and would like to help, take the quiz here.
18 comments:
Dear Eric,
I started taking the quiz, and fell at the first hurdle.
I thought that I was going to be asked whether responses to your questionnaire were written by DD or by GPT-3.
But instead, I was asked whether responses were "like what DD might say." That's a different issue.
Consider the response that says that he's working on a book called "Freedom Evolves." Even a cursory knowledge.of DD's bib shows that this cannot have been a response that DD gave to your recent questionnaire, and must have been written by GPT-3.
But is it "like what DD might say?" Sure! Exactly like what he might say, because he probably did say it in 2002 or so.
So, if the question is, "DD or GPT-3?", then the answer is clearly "GPT-3." But asking about whether a passage is "like something DD might say" is asking something that has to be answered "yes" for anything that DD did in fact say, at any time in his life.
Which question should I answer?
Did I take the survey...
...it seem to have ended, for me, with an arrow pointing to the right (for more survey?), I clicked on the arrow, but my computer could not get beyond the arrow, is that it?
Distinguishing: My and I perceptions with sense and feel perceptions...
Anonymous & Arnold
The survey has two parts:
(1) Choose the answer you think Dennett actually gave (by clicking it);
(2) Rate the "Dennett-likeness" of each proposed answer.
If you only rate the 5-point "Dennett-likeness" but don't choose one of the answers at the top of the page as Dennett's actual answer, you can't proceed with the survey (I couldn't).
Yes, these are different issues.
There's only one correct choice regarding (1).
There's ... room for debate regarding (2).
... and yes, I initially did fail to read the instructions properly and did wonder why I was not able to proceed ...
Thanks! I'm Anonymous #1 up there, and this helps to answer my question.
There is still a smaller problem with the second task, roughly about temporal indexing. There are things that sound like something DD might have said *long ago*, which do not sound like things he might have said *recently* In response to ES's questionnaire, eg the line about his upcoming 2003 book.
You've helped, tho, and progress is good.
Thanks all! Yes, Anon #1, although Anon #2 is correct about the structure of the survey, you're right about the temporal indexing issue. Arnold: Sorry for the confusing bit at the end, redirecting to Prolific. You can ignore that!
On Facebook, some people have indicated that they have been told "how they did". Others (me, David Chalmers) had not been told (and I haven't been told though I gave my email address). Wondering if this is automated?
Michael Kremer
Anon2, Eric, got it thanks....
Very much looking forward to seeing "how I did," because I developed many views about the characteristics of genuine Dennett prose from this exercise, so it would be nice to know if my basic facts are right before I proceed to the theories.
0/10. Well done GPT-3!
Some of these imitations are astonishingly compelling! In some cases, my strategy was to look for imperfections to try and identify the human author...
@Daniel Hoek
So part of your strategy is to assume that Dennett is perfect and the AI is not? Or that the AI is perfect and Dennett is not?
6/10, not proud of it but at least I am a weakly reliable Dennett Detector.
How did you get your score? Do you also know which ones you got right? Is this info somewhere else on the web?
When you say finetuned 'GPT-3', do you mean you used the finetuning part of the OA API for GPT-3-175b (which is what most people assume 'GPT-3' means) or did you actually do something else like finetune the much smaller GPT-J or GPT-Neo or something?
I got it per mail several hours after I participated. The correct answers will be revealed, it said, once the Experiment concluded.
I got 7/10, which shows that I'm a perfect Dennett Detector.
(Since there were three times when Dennett just wasn't sounding like himself.)
Thanks for taking the test and for the comments!
To those waiting on scores: Right, I have been viewed the scores then sending the reply emails with the info. That should all be done now, so email me if you still haven't received your score.
Gwern: Matt Crosby is our programming expert who did the fine tuning. I'll check with him about the exact model, but I do know that it was Davinci with four epochs.
Your description says: "So we've asked Dennett ten philosophical questions and recorded his answers. We posed the same questions to GPT-3, four times for each of the ten questions, to get four different answers for each question. ... We'd love it if you can take a quiz to see if you can pick out Dennett's actual answer to each question."
I'd like to ask if this is strictly the case. That is, are the texts presented either the *entire* answer offered by Dennett or GPT-3 to your question (on the occasion) or *excerpts* from those answers, and if the latter how were the excerpts chosen?
Very impressive. I wasn't sure on several of the questions. I remember a science fiction story in which the job of a (commercial) writer had become not drafting, but selecting from a bunch of good options offered by a computer auto-drafter. Maybe we'll get there soon!
Post a Comment