I'm an introspective pessimist: I think people's introspective judgments about their stream of conscious experience are often radically mistaken. This is the topic of my 2011 book, Perplexities of Consciousness. Over the past few years, I've found the most common objection to my work to be what I'll call the Individual Differences Reply.
Many of my arguments depend on what I call the Argument from Variability: Some people report experiences of Type X, others report experience of Type not-X, but it's not plausible that Group 1 and Group 2 differ enough in their underlying stream of experience to make both claims true. Therefore, someone must be mistaken about their experience. So for example: People used to say they dreamed in black and white; now people don't say that; so someone is wrong about their dream experiences. Some people say that tilted coins in some sense look elliptical as though projected upon a 2D visual screen, while others say that visual experience is robustly 3D with no projective distortions at all. Some people say that objects seen 20 degrees from the point of fixation look fairly clear and colorful, while others say objects 20 degrees from center look extremely sketchy and indeterminate of color. Assuming a common underlying experience, disagreement reveals someone to be mistaken. (Fortunately, we needn't settle which is the mistaken party.)
The Individual Differences Reply challenges this assumption of underlying commonality. If people differ in their experience as radically as is suggested by their introspective reports, then no one need be mistaken! Maybe people did really used to dream in black and white and now they dream in color (see Ned Block's comment on a recent Splintered Mind post). Maybe some people really do see tilted coins as elliptical while others do not.
There are two versions of the Individual Difference Reply, which I will call the Stable Differences and the Temporary Differences versions. On the Stable Differences version, people durably differ in their experiences in such a way as to render their reports accurate as generalizations about their experiences. On the Temporary Differences version, people might be similar in their experiences generally, but when faced with the introspective task itself their experience shifts in such a way as to match their reports. For example, maybe everyone generally has similar experiences 20 degrees into the visual periphery, but when asked to introspect their visual experience, some people experience (and accurately report) clarity while others experience (and accurately report) haziness.
Here's how I think the dispute should be resolved: Look, on a case-by-case basis, for measurable brain or behavioral differences that tightly correlate with the differences in introspective report and which are plausibly diagnostic of the underlying variability. If those correlations are found, accept that the reports reveal real differences. If not, conclude that at least one party is wrong about their experience.
Consider black and white dreaming. Theoretically, we could hook up brain imagining machinery to people who report black and white dreams and to people who report color dreams. If the latter group shows substantially more neural activity in color-associated neural areas while dreaming, the reports are substantiated. If not, the reports are undermined. Alternatively, examine color term usage in dream narratives: How often do people use terms like "red", "green", etc., in dream diaries? If the rates are different, that supports the existence of difference; if not, that supports the hypothesis of error. In fact, I looked at exactly this in Chapter 1 of Perplexities, and found rates of color-term usage in dream reports to be the same during the peak period of black-and-white dream reporting (USA circa 1950) and recently (USA circa 2000).
Generally, I think the evidence often shows a poor relationship between differences in introspective report and differences in plausibly corresponding behavior or physiology.
Sometimes there is no systematic evidence, but antecedent plausibility considerations suggest against at least the Stable Differences version of the Individual Differences Reply: It seems highly unlikely that the people who report sharp visual experiences 20 degrees into the periphery differ vastly in general visual capacity or visual physiology from those who report highly indeterminate shape and color 20 degrees into the periphery. But that's an empirical question, so test it if you like!