What is "Validity" in Social Science? Validity As a Property of Inferences vs of Claims

If you want to annoy your psychology and social science friends, I have just the trick!  Gather four of them together and ask them to explain exactly what validity is.  Then step back and watch them descend into confusion and contradiction.  Bring snacks.

We use the term all the time, with a truly bewildering array of modifiers: internal validity, construct validity, content validity, external validity, logical validity, statistical conclusion validity, discriminant validity, convergent validity, face validity, criterion validity....  Is there one thing, validity in general, which undergirds all of these uses?  And if so, what does it amount to?  Or is "validity" more of a family resemblance concept?  Are all true statements in some sense valid?  Or is validity more specific than that -- perhaps a matter of appropriate application of method?  Can a study or a conclusion or a method or an instrument be valid even if it's entirely mistaken, as long as proper techniques have been employed?  Oh, and wait, is validity really a property of studies and conclusions and methods and instruments?  They seem so different and to have such different criteria of success!

[image: A Defence of the Validity of the English Ordinations]

I've found surprisingly few general treatments of validity in the social sciences which articulate the concept with the kind of rigor and consistency that would satisfy an analytic philosopher.  One of the best and most influential recent attempts is Shadish, Cook, and Campbell 2002.  I'm going to poke at their treatment with one question in mind: What is validity a property of?

Shadish, Cook, and Campbell begin with a seemingly clear commitment: validity is a property of inferences:

We use the term validity to refer to the approximate truth of an inference.[1]  When we say something is valid, we make a judgment about the extent to which relevant evidence supports the inference as being true or correct (p. 34).

In the next paragraph, they emphasize again that validity is a property of specifically of inferences:

Validity is a property of inferences.  It is not a property of designs or methods, for the same design may contribute to more or less valid inferences under different circumstances....  So it is wrong to say that a randomized experiment is internally valid or has internal validity -- although we may occasionally speak that way for convenience (p. 34).

Characterizing validity as a property of inferences resonates with the use of "validity" in formal logic, where it is also generally treated as a property of deductive inferences (well, more accurately, a property of deductive arguments -- but close enough, if we treat inferences as psychological instantiations of arguments).  In formal logic, an inference or argument is deductively valid if and only if, in virtue of its form, it's impossible for the conclusion of the inference to be false if the premises of the inference are true.  [Okay, fine, maybe it's not that simple, but let's not go there today.]

Consider, for example, modus ponens, the inference form in which "P" and "If P, then Q" serve as premises, and "Q" serves as the conclusion.  (P and Q are propositions.)  Modus ponens is normally viewed as a valid form of inference because under the assumption that the two premises are true, the conclusion must be true.  If it's true that Socrates is a man and also true that If Socrates is a man, then Socrates is mortal, then it must also be true that Socrates is mortal.

Logicians normally distinguish validity from soundness: An inference is sound if and only if the inference is valid and the premises are true.  An inference can of course be valid without being sound, for example: (P1.) I am wearing three hats.  (P2.) If I am wearing three hats, I am a famous actor.  (C.) Therefore, I am a famous actor.  That's a perfectly valid inference to a perfectly false conclusion (thanks to at least one false premise).

Inferences are not true or false.  They are valid or invalid.  What is true or false are propositions: the premises and the conclusion.  Got it?  Good!  Lovely!  Now let's go back for a closer look at Shadish et al.  This time let's not forget footnote 1.

We use the term validity to refer to the approximate truth of an inference.[1]  When we say something is valid, we make a judgment about the extent to which relevant evidence supports the inference as being true or correct.

[1] We might use the terms knowledge claim or proposition in place of inference here, the former being observable embodiments of inferences.  There are differences implied by each of these terms, but we treat them interchangeably.

Okay, now wait.  Is validity a property of an inference or is it a property of a claim or proposition?  An inference is one thing and a claim is another!  Shadish et al., despite emphasizing that validity is a property of inferences, confusingly add they will treat "inference" and "knowledge claim" interchangeably.  But an inference is not a knowledge claim.  An inference is a process of moving from the hypothesized truth of one or more claims to a conclusion which, if all goes well, is true if the claims are true.

Could we maybe just say that validity is a property of an inference that has a true conclusion at the end, as a result of employing of good methods?  (This would make "validity" in Shadish et al.’s sense closer to "soundness" in the logician’s sense.)  Or differently but relatedly could we say that validity is a property that a claim has when it is both true and the result of methodologically good inference (and where the truth and inference quality are non-accidentally related)?  Or is validity about justification rather than truth -- "the extent to which relevant evidence supports the inference as being true or correct" (italics added).  Justification can of course diverge from the truth, since sometimes evidence strongly supports a proposition that turns out to be false in the end.  Or should we go back to process here, as suggested by the term "correct", since presumably an inference can be correct, in the sense that it is the right inference to make given the evidence, without its conclusion being true?

Oy vey.  I wish I could say that Shadish et al. clarify this all later and use their terms consistently throughout their influential book, but that's not so -- as indeed they hint in their remark, quoted above, about sometimes speaking loosely as though experiments (and not just inferences or claims) can be valid.  Their book is a lovely guide to empirical methods, but by the standards of analytic philosophy their definition of validity is a mess.

But this post isn't just about Shadish et al. (despite their 47,473 citations as of today).  It's about the treatment of validity in psychology and the social sciences in general.  Shadish et al. exemplify a conceptual looseness I see almost everywhere.

As a first-pass corrective on this looseness let me propose the following:

Psychologists' and social scientists' claims about validity, in my judgment, make the most sense on the whole and are simplest to interpret if we treat validity as fundamentally a property of claims or propositions rather than as a property of inferences (or methods or instruments or experiments).  A causal generalization, for example, of the form that events of type A cause events of type B in conditions C is "valid" if and only if events of type A do cause events of type B in conditions C.  To say that a psychological instrument (such as an IQ test) is "valid" is fundamentally matter of saying that the instrument measures what it claims to measure: Validity is a matter of the truth of that claim.  A study is valid if the claims of which it is composed are true (both its claims about its conclusions and its claims about the manner in which its conclusions are supported).  A measure has "face validity" if superficially it looks like the claims that result from applying that measure will be true claims.  Two measures have "discriminant validity" if the following claim is true: They in fact measure different underlying phenomena.

Validity, in the psychologists' and social scientists' sense, is best conceptualized as a property that belongs to claims: the property those claims have when they are true.  Attributions of validity to ontological entities other than claims, such as measures and studies, can all be reinterpreted as commitments to the truth of certain types of claims that are implicitly or explicitly embodied in the application of measures, the publication of studies, the making of inferences, etc.  (That good method has been used to arrive at the claims, I regard as a cancelable implicature.)

Why go this direction?  If we treat "validity" as a matter of the quality of the inference or the degree of justification of the conclusion regardless of whether the conclusion is in fact true, then we will have a plethora of valid inferences and valid conclusions, and by extension valid measures, valid instruments, and valid causal models that are completely mistaken, because science is hard and what you're justified in concluding is often not so.  But that's not how social scientists generally talk: A valid measure is one that is right, one that works, one that measures what it's supposed to measure, not one that we are (perhaps falsely) justified in thinking is right.

I diagnose the confusion as arising from three sources: First, widespread sloppy conceptual practice that uses "valid" loosely as a general term of praise.  Second, a tendency among those who do want to rigorize to notice that the philosophers' logical notion of validity applies to arguments or inferences, and consequently some corresponding pressure to think of it that way in the social sciences too, despite the dominant grain of social science usage running a different direction.  Third, a confusing liberality both about the types of validity and the ontological objects that can be said to have validity, which makes it hard to see the simple core underlying idea behind it all: that validity is nothing but a fancy word for truth.