Adversarial Collaboration

[originally posted at Brains Blog, with a lovely reply by Justin Sytsma, in which he compares your mind to Emmenthaler cheese]

You believe P. Your opponent believes not-P. Each of you thinks that new empirical evidence, if collected in the right way, will support your view. Maybe you should collaborate? An adversary can keep you honest and help you see the gaps and biases in your arguments. Adversarial collaboration can also add credibility, since readers can’t as easily complain about experimenter bias. Plus, when the data land your way, your adversary can’t as easily say that the experiment was done wrong!

My own experience with adversarial collaboration has been mostly positive. From 2004-2011, I collaborated with Russ Hurlburt on experience sampling methods (he’s an advocate, I’m a skeptic). Since 2017, I’ve been collaborating with Brad Cokelet and Peter Singer on whether teaching meat ethics to university students influences their campus food purchases (they thought it would, while I was doubtful). The first collaboration culminated in a book with MIT Press and double-issue symposium in Journal of Consciousness Studies. The second has so far produced an article in Cognition and hopefully more to come. Other work has been partly adversarial or conducted with researchers whose empirical guesses differed from mine.

I’ve also had two adversarial collaborations fail – fortunately in the early stages. Both failed for the same reason: lack of well-defined common ground. Securing common ground is essential to publication and uniquely challenging in adversarial collaboration.

I have three main pieces of advice:

(1.) Choose a partner who thrives on open dialogue.

(2.) Define your methods early in the project, especially the means of collecting the crucial data.

(3.) Segregate your empirical results from your theoretical conclusions.

To publish anything, you and your co-authors must speak as one. Without open dialogue, clearly defined methods, and segregation of results from theory, adversarial projects risk slipping into irreconcilable disagreement.

Open Dialogue

In what Jon Ellis and I have called open dialogue, you aim to present not just arguments in support of your position P but your real reasons for holding the view you hold, inviting scrutiny not only of P but also of the particular considerations you find convincing. You say “here’s why I think that P” with the goal of offering considerations C1, C2, and C3 in favor of P, where C1-3 (a.) epistemically support P and also (b.) causally sustain your opinion that P. Instead of having only one way to prove you wrong – showing that P is false or unsupported – your interlocutor now has three ways to prove you wrong. They can show P to be false or unsupported; they can show C1-3 to be false or unsupported; or they can show that C1-3 don’t in fact adequately support P. If they meet the challenge, your mind will change.

Contrast the lawyerly approach, the approach of someone who only aims to convince you or some other audience (or themselves, in post-hoc rationalization). The lawyerly interlocutor will normally offer reasons in favor of P, but if those reasons are defeated, that’s only a temporary inconvenience. They’ll just shift to a new set of reasons, if new reasons can be found. And in complicated matters of philosophy and human science, people can almost always find multiple reasons not to reject their pet ideas if they’re motivated enough. This can be frustrating for partners who had expected open dialogue! The lawyer’s position has, so to speak, secret layers of armor – new reasons they’ll suddenly devise if their first reasons are defeated. The open interlocutor, in contrast, aims to reveal exactly where the chinks in their armor are. They present their vulnerabilities: C1-3 are exactly the places to poke at if you want to win them over. Their opinion could shift, and such-and-such is what it would take.

In empirical adversarial collaboration, the most straightforward place to find common ground is in agreement that some C1 is a good test of P. You and your adversary both agree that if C1 proves to be empirically false, belief in P ought to be reduced or withdrawn, and if C1 proves to be empirically true, P is supported. Without open dialogue, you cannot know where your adversary’s reasoning rests. You can’t rely on the common ground that C1 is a good test of P. You thought you were testing P by means of testing C1. You thought that if C1 failed, your adversary would withdraw their commitment to P and you could write that up as your mutual result. If your adversary instead shifts lawyerlike to a new C2, the common ground you thought you had, the theoretical core you thought you shared, has disappeared, and your project has surprisingly changed shape. In one failed collaboration, I thought my adversary and I had agreed that such-and-such empirical evidence (from one of their earlier unpublished studies) wasn’t a good test of P, and so we began piloting alternative tests. However, they were secretly continuing to collect data on that earlier study. With the new data, their p value crossed .05, they got a quick journal acceptance – and voilà, they no longer felt that further evidence was necessary.

Now of course we all believe things for multiple reasons. Sometimes when new evidence arrives we find that our confidence in P doesn’t shift as much as we thought it would. This can’t be entirely known in advance, and it would be foolish to be too rigid. Still, we all have the experience of collaborators and conversation partners who are more versus less open. Choose an open one.

Define Your Methods Early

If C1, then P; and if not-C1 then not-P. Let’s suppose that this is your common ground. One of you thinks that you’ll discover C1 and P will be supported; the other thinks that you’ll discover the falsity of C1 and P will be disconfirmed. Relatively early in your collaboration, you need to find a mutually agreeable C1 that is diagnostic of the truth of P. If you’re thinking C1 is the way to test P and C2 wouldn’t really show much, while your adversary thinks C2 is really more diagnostic, you won’t get far. It’s not enough to disagree about the truth of P while aiming in sincere fellowship to find a good empirical test. You must also agree on what a good test would be – ideally a test in which either a positive or a negative result would be interesting. An actual test you can actually run! The more detailed, concrete, and specific, the better. My other failed collaboration collapsed for this reason. Discussion slowly revealed that the general approach one of us preferred was never going to satisfy the other two.

If you’re unusually lucky, maybe you and your adversary can agree on an experimental design, run the experiment, and get clean, interpretable results that you both agree show that P. It worked, wow! Your adversary saw the evidence and changed their mind.

In reality of course, testing is messy, results are ambiguous, and after the fact you’ll both think of things you could have done better or alternative interpretations you’d previously disregarded – especially if the test doesn’t turn out as you expected. Thinking clearly in advance about concrete methods and how you and your adversary would interpret alternative results will help reduce, but probably won’t eliminate, this shifting.

Segregate Your Empirical Results from Your Theoretical Conclusions

If you and your adversary choose your methods early and favor an open rather than a lawyerly approach, you’ll hopefully find yourselves agreeing, after the data are collected, that the results do at least superficially tend to support (or undermine) P. One of you is presumably somewhat surprised. Here’s my prediction: You’ll nevertheless still disagree about what exactly the research shows. How securely can you really conclude P? What alternative explanations remain open? What mechanism is most plausibly at work?

It’s fine to disagree here. Expect it! You entered with different understandings of the previous theoretical and empirical literature. You have different general perspectives, different senses of how considerations weigh against each other. Presumably that’s why you began as adversaries. That’s not all going to evaporate. My successful collaborations were successful in part, I think, because we were unsurprised by continuing disagreement and thus unphased when it occurred, even though we were unable to predict in advance the precise shape of our evolving thoughts.

In write-up, you and your adversary will speak with one voice about motivations, methods, and results. But allow yourself room to disagree in the conclusion. Every experiment in the human sciences admits of multiple interpretations. If you insist on complete theoretical agreement, your project might collapse at this last stage. For example, the partner who is surprised the by results might insist on more follow-up studies than is realistic before they are fully convinced.

Science is hard. Science with an adversary is doubly hard, since sufficient common ground can be difficult to find. However, if you and your partner engage in open dialogue, the common ground is less likely to suddenly shift away than if one or both of you prevaricate. Early specification of methods helps solidify the ground before you invest too heavily in a project doomed by divergent empirical approaches. And allowing space at the end for alternative interpretations serves as a release valve, so you can complete the project despite continuing disagreement.

In a good adversarial collaboration, if you win you win. But if you lose, you also win. You’ve shown something new and (at least to you) surprising. Plus, you get to parade your virtuous susceptibility to evidence by uttering those rare and awesome words, “I was wrong.”

[image source]