Thursday, August 27, 2020

What is "Validity" in Social Science? Validity As a Property of Inferences vs of Claims

If you want to annoy your psychology and social science friends, I have just the trick!  Gather four of them together and ask them to explain exactly what validity is.  Then step back and watch them descend into confusion and contradiction.  Bring snacks.

We use the term all the time, with a truly bewildering array of modifiers: internal validity, construct validity, content validity, external validity, logical validity, statistical conclusion validity, discriminant validity, convergent validity, face validity, criterion validity....  Is there one thing, validity in general, which undergirds all of these uses?  And if so, what does it amount to?  Or is "validity" more of a family resemblance concept?  Are all true statements in some sense valid?  Or is validity more specific than that -- perhaps a matter of appropriate application of method?  Can a study or a conclusion or a method or an instrument be valid even if it's entirely mistaken, as long as proper techniques have been employed?  Oh, and wait, is validity really a property of studies and conclusions and methods and instruments?  They seem so different and to have such different criteria of success!

[image: A Defence of the Validity of the English Ordinations]

I've found surprisingly few general treatments of validity in the social sciences which articulate the concept with the kind of rigor and consistency that would satisfy an analytic philosopher.  One of the best and most influential recent attempts is Shadish, Cook, and Campbell 2002.  I'm going to poke at their treatment with one question in mind: What is validity a property of?

Shadish, Cook, and Campbell begin with a seemingly clear commitment: validity is a property of inferences:

We use the term validity to refer to the approximate truth of an inference.[1]  When we say something is valid, we make a judgment about the extent to which relevant evidence supports the inference as being true or correct (p. 34).

In the next paragraph, they emphasize again that validity is a property of specifically of inferences:

Validity is a property of inferences.  It is not a property of designs or methods, for the same design may contribute to more or less valid inferences under different circumstances....  So it is wrong to say that a randomized experiment is internally valid or has internal validity -- although we may occasionally speak that way for convenience (p. 34).

Characterizing validity as a property of inferences resonates with the use of "validity" in formal logic, where it is also generally treated as a property of deductive inferences (well, more accurately, a property of deductive arguments -- but close enough, if we treat inferences as psychological instantiations of arguments).  In formal logic, an inference or argument is deductively valid if and only if, in virtue of its form, it's impossible for the conclusion of the inference to be false if the premises of the inference are true.  [Okay, fine, maybe it's not that simple, but let's not go there today.]

Consider, for example, modus ponens, the inference form in which "P" and "If P, then Q" serve as premises, and "Q" serves as the conclusion.  (P and Q are propositions.)  Modus ponens is normally viewed as a valid form of inference because under the assumption that the two premises are true, the conclusion must be true.  If it's true that Socrates is a man and also true that If Socrates is a man, then Socrates is mortal, then it must also be true that Socrates is mortal.

Logicians normally distinguish validity from soundness: An inference is sound if and only if the inference is valid and the premises are true.  An inference can of course be valid without being sound, for example: (P1.) I am wearing three hats.  (P2.) If I am wearing three hats, I am a famous actor.  (C.) Therefore, I am a famous actor.  That's a perfectly valid inference to a perfectly false conclusion (thanks to at least one false premise).

Inferences are not true or false.  They are valid or invalid.  What is true or false are propositions: the premises and the conclusion.  Got it?  Good!  Lovely!  Now let's go back for a closer look at Shadish et al.  This time let's not forget footnote 1.

We use the term validity to refer to the approximate truth of an inference.[1]  When we say something is valid, we make a judgment about the extent to which relevant evidence supports the inference as being true or correct.

[1] We might use the terms knowledge claim or proposition in place of inference here, the former being observable embodiments of inferences.  There are differences implied by each of these terms, but we treat them interchangeably.

Okay, now wait.  Is validity a property of an inference or is it a property of a claim or proposition?  An inference is one thing and a claim is another!  Shadish et al., despite emphasizing that validity is a property of inferences, confusingly add they will treat "inference" and "knowledge claim" interchangeably.  But an inference is not a knowledge claim.  An inference is a process of moving from the hypothesized truth of one or more claims to a conclusion which, if all goes well, is true if the claims are true.

Could we maybe just say that validity is a property of an inference that has a true conclusion at the end, as a result of employing of good methods?  (This would make "validity" in Shadish et al.’s sense closer to "soundness" in the logician’s sense.)  Or differently but relatedly could we say that validity is a property that a claim has when it is both true and the result of methodologically good inference (and where the truth and inference quality are non-accidentally related)?  Or is validity about justification rather than truth -- "the extent to which relevant evidence supports the inference as being true or correct" (italics added).  Justification can of course diverge from the truth, since sometimes evidence strongly supports a proposition that turns out to be false in the end.  Or should we go back to process here, as suggested by the term "correct", since presumably an inference can be correct, in the sense that it is the right inference to make given the evidence, without its conclusion being true?

Oy vey.  I wish I could say that Shadish et al. clarify this all later and use their terms consistently throughout their influential book, but that's not so -- as indeed they hint in their remark, quoted above, about sometimes speaking loosely as though experiments (and not just inferences or claims) can be valid.  Their book is a lovely guide to empirical methods, but by the standards of analytic philosophy their definition of validity is a mess.

But this post isn't just about Shadish et al. (despite their 47,473 citations as of today).  It's about the treatment of validity in psychology and the social sciences in general.  Shadish et al. exemplify a conceptual looseness I see almost everywhere.

As a first-pass corrective on this looseness let me propose the following:

Psychologists' and social scientists' claims about validity, in my judgment, make the most sense on the whole and are simplest to interpret if we treat validity as fundamentally a property of claims or propositions rather than as a property of inferences (or methods or instruments or experiments).  A causal generalization, for example, of the form that events of type A cause events of type B in conditions C is "valid" if and only if events of type A do cause events of type B in conditions C.  To say that a psychological instrument (such as an IQ test) is "valid" is fundamentally matter of saying that the instrument measures what it claims to measure: Validity is a matter of the truth of that claim.  A study is valid if the claims of which it is composed are true (both its claims about its conclusions and its claims about the manner in which its conclusions are supported).  A measure has "face validity" if superficially it looks like the claims that result from applying that measure will be true claims.  Two measures have "discriminant validity" if the following claim is true: They in fact measure different underlying phenomena.

Validity, in the psychologists' and social scientists' sense, is best conceptualized as a property that belongs to claims: the property those claims have when they are true.  Attributions of validity to ontological entities other than claims, such as measures and studies, can all be reinterpreted as commitments to the truth of certain types of claims that are implicitly or explicitly embodied in the application of measures, the publication of studies, the making of inferences, etc.  (That good method has been used to arrive at the claims, I regard as a cancelable implicature.)

Why go this direction?  If we treat "validity" as a matter of the quality of the inference or the degree of justification of the conclusion regardless of whether the conclusion is in fact true, then we will have a plethora of valid inferences and valid conclusions, and by extension valid measures, valid instruments, and valid causal models that are completely mistaken, because science is hard and what you're justified in concluding is often not so.  But that's not how social scientists generally talk: A valid measure is one that is right, one that works, one that measures what it's supposed to measure, not one that we are (perhaps falsely) justified in thinking is right.

I diagnose the confusion as arising from three sources: First, widespread sloppy conceptual practice that uses "valid" loosely as a general term of praise.  Second, a tendency among those who do want to rigorize to notice that the philosophers' logical notion of validity applies to arguments or inferences, and consequently some corresponding pressure to think of it that way in the social sciences too, despite the dominant grain of social science usage running a different direction.  Third, a confusing liberality both about the types of validity and the ontological objects that can be said to have validity, which makes it hard to see the simple core underlying idea behind it all: that validity is nothing but a fancy word for truth.


Stephen Wysong said...

I can't recall where I read the remark that every social science is "a body of knowledge in search of a discipline," which seems a valid observation ... ;-)

Callan said...

I don't think so, Eric. Science as a practice is loathe to actually say it has proven anything - proven/truth is a place where the scientific angels fear to tread. It's down to mortals to make the conclusion for themselves individually in the end. Validity is just trying to prompt someone else to accept an idea without the first person actually just saying 'I'm right, dammit!'. Kind of like a Wayne's world line "Person who agrees on the external validity of my experiment says what?". It's probably a little passive aggressive.

David Duffy said...

I agree that the logician's validity is not appropriate. I think the feature that applies to most of the disparate (face, criterion etc) types is they involve (inductive or abductive) inference that the conclusions are correct outside of the terms of the study as straight statistical hypothesis test. So, for example, criticism of the choice of statistical methodology as inappropriate to the question is an attack on validity, but doesn't question the calculations used in the published analysis. I would almost say validity is soundness, except that some attacks on validity are meta-cum-holistic, while others, as in the case of discriminant validity are internal but oblique to the "main line of argument".

Eric Schwitzgebel said...

Stephen: A bit harsh, I think, despite having *some* truth!

Callan: What you suggest is part of the rhetoric of science and probably part of the explanation of why scientists often prefer obscure terms like "valid" when they probably really mean "true". I don't think that that way of thinking about science stands us very well to epistemic scrutiny unless we just want to remove useful terms like "true" and "know" from our vocabulary. Scientists should be willing to say that X is true and they know it (under appropriate conditions, of course), while acknowledging of course that they're always a chance they are mistaken.

David: That seems right. After some comments here and on social media and some further thoughts of my own, I am now leaning toward a broader conception of "valid" as meaning "scientifically correct", where truth is one important type (but not the only important type) of scientific correctness.

Callan said...

Eric: What would you say is the difference between 'The evidence suggests X is the case' and 'X is true, but I could be mistaken' is? Usually there's some kind of p(robability) value in the statistical findings that indicates that the findings could have a chance of actually being incorrect. Do you feel the scientist should be owning that instead and saying they could be mistaken rather than saying the evidence could be incorrect in what it suggests?

Off topic PS: Recently ran into someone who said all interaction between individuals is manipulation. Wondered what your take on that is?

Eric Schwitzgebel said...

I think it's probably fine to say it either way. I hope I'm not committed to seeing one or the other of those two ways of speaking as wrong!

On interaction as manipulation: It's hard to see either an ordinary language or a pragmatic-stipulative definition of "manipulation" on which that would be true. What's the point of talking that way rather than distinguishing between manipulative and non-manipulative interactions on grounds of important differences between types of interaction?

Callan said...

Eric, I think an issue in human thought is putting ones own credibility on the line in saying something - it kind of biases us towards confirmation bias, because if we are wrong we are not just wrong but we lose credibility. That might be why there is a shift from 'This is true' to 'this has validity'. It avoids an issue that prompts people to try to avoid being potentially wrong. Risking being wrong is, I would say, an important part of critical thinking. When you say 'X is true' but then it turns out you're wrong, that can be hard to take and so it's hard to try and actually look for ways you are wrong - even with the 'I could be mistaken' part added. Because it's 'I' could, which kind of suggests it's not something that is wrong with the idea but something that is wrong with you. That's harder to take. So that's my guesstimate on it, I could be...waitasecond...the idea could be wrong!

On manipulation, yes I wondered what the point was as well. Like I gave the example of where cookoo bird babies reside in the nest of other birds and take food from those parent birds Vs where the parent birds feed their own baby birds - you can't call both 'manipulation' as much as each other, can you? You'd at least want to say 'parasitic manipulation' vs 'symbiotic manipulation' or something to distinguish them. I was trying to find some articles on the matter of 'what is and isn't manipulation' and didn't get far, strangely. If you can think of any off the top of your head and can mention them I'd appreciate it :)

Anonymous said...

I recommend reading Denny Borsboom's work. There is this 2004 paper "The concept of validity" from Psych Review:

as well as a couple of books

Here's a link to his home page

He's the best philosopher of psychometrics . . . for whatever that's worth.

--Rob Chametzky

Eric Schwitzgebel said...

Thanks, Rob! I know his classic 2004 article but I confess I haven't yet made my way through his books.

Unknown said...

I came here looking for anyone else who has experienced my "seeing red" moment, though like only a couple mentions above, I did not see red, I saw grey. It was in an intense exercise in my acting class in university, and my scene partner was driving me absolutely livid. Long story short, I blinked and my vision was suddenly a faded grey, and it was like a veil or filter or something just came over my eyes. I felt distant from her, kind of like I was looking through a lens, but my body felt as present as ever. I don't ever recall having this experience before, but I was so alarmed when it did happen; all the colour in the room simply vanished from my sight. It ended when I controlled my temper, but I remember it so vividly that I keep wondering how and why it happened. I've had no experience seeing red, literally, but only this.