Kyle Thompson (who recently earned his PhD under my supervision) has published the central findings of a dissertation that raises exactly this challenge to experimental philosophy. His approach is to compare the standard quantitative measures of participants' opinions -- that is, participants' numerical responses on standardized questions -- with two qualitative measures: what participants say when instructed to "think aloud" about the experimental stimuli and a post-response interview about why they answered the way they did.
Kyle's main experiment replicates the quantitative results of an influential study that purports to show that ordinary research participants reject the "ought implies can" principle. According to the ought-implies-can principle, people can only be morally required to do what it is possible for them to do. Thompson replicates the quantitative results of the earlier experiment, seeming to confirm that participants reject ought-implies-can. However, Thompson's qualitative think-aloud and interview results clearly indicate that his participants actually accept, rather than reject, the principle. The quantitative and the qualitative results point in opposite directions, and the qualitative results are more convincing.
In the scenario of central interest, "Brown" agrees to meet a friend at a movie theater at 6:00. But then
As Brown gets ready to leave at 5:45, he decides he really doesn't want to see the movie after all. He passes the time for five minutes, so that he will be unable to make it to the cinema on time. Because Brown decided to wait, Brown can't meet his friend Adams at the movie by 6.
Participants then rate their degree of agreement or disagreement with the following three questions:
At 5:50, Brown can make it to the theater by 6 Brown is to blame for not making it to the theater by 6 Brown ought to make it to the theater by 6
As you might expect, in both the original article and Thompson's replication, participants almost all disagree that Brown can make it to the theater by 6. So far, so good. However, apparently in violation of the ought-implies-can principle, participants overall tended to agree that Brown is to blame for not making it to the theater by 6 and (to a lesser extent) that Brown ought to make it to the theater by 6. Interpreting the results at face value, it appears that regarding making it to the theater by 6, participants think that Brown cannot do it, that he is blameworthy for not doing it, and that he ought to do it -- and thus that someone can be blameworthy for failing to do, and ought to do, something that it is not possible for them to do.
Now, if your reaction to this is wait a minute..., you share something in common with Thompson and me. Participants' think-aloud statements and subsequent interviews reveal that almost all of them reinterpret the questions to preserve consistency with the ought-implies-can principle. For example, some participants explain their positive answers to "Brown ought to make it to the theater by 6" by explaining that Brown ought to try to make it to the theater by 6. Others change the tense and the time referent, explaining that Brown "could have" made it to the theater and that he should have left by 5:45. There is no violation of ought-implies-can in either response. At 5:50, Brown could presumably still try to make it to the theater. And at 5:45 he still could have made it to the theater.
Through careful examination of the transcripts, Thompson discovers that the almost 90% of participants in fact adhere to the ought-implies-can principle in their responses, often reinterpreting the content or tense of the questions to render them consistent with this principle.
As far as I'm aware, this is the first attempt to replicate a quantitative experimental philosophy study with careful qualitative interview methods. What it suggests is that the surface-level interpretation of the quantitative results can be highly misleading. The majority of participants appear to have the opposite of the view suggested by their quantitative answers.
It is an open question how much of the quantitative research in experimental philosophy would survive careful qualitative scrutiny. I hope others follow in Kyle's footsteps by attempting careful qualitative replications of important quantitative work in the subdiscipline.