... if your aim is to encourage students to actually act better. (Of course, this might not be among your aims.)
(This post was inspired by Janet Stemwedel's Facebook/Twitter post about students cheating in her ethics class, and subsequent discussion.)
1. Creating the appearance that every pro has an equally good con. Annette Baier is among those who have emphasized this risk. A typical philosophical teaching style is to present both sides of every major topic discussed, in a more-or-less even-handed way. We tell students that they are welcome to defend either the pro or the con, and we encourage contrarian students who challenge the reasoning and conclusions of the assigned authors. This even-handed debate-like format might lead some students to think that, in ethical reasoning, there are no right or wrong answers to be found, just interminable back and forth. Probably this attitude will have little effect on students' practical choices outside of the classroom; but if it does have an effect, it might be to weaken their sense that ethical principles that they might otherwise have acted on are as sound and indisputable as they would previously have thought.
2. Improving one's skill at moral rationalization. Suppose you want to do X -- steal a library book, for example. Of course, you wouldn't do that. It's wrong! But wait. Remember that ethics class you took? Maybe you can construct a utilitarian defense of stealing the book. No one would miss it that much, and you'd benefit greatly from keeping it. The institution has much more money than you do, and can easily replace it. Stealing the book would maximize human happiness! (Especially yours.) Such reasoning is rationalization, in the pejorative sense of that term, if your reasoning is basically just a biased search for reasons in support of the self-serving conclusion you'd like to reach. If you're tempted to do something morally wrong, skill at philosophical reasoning, and knowledge of a diverse range of possibly relevant moral principles, might enable you to better construct superficially attractive arguments that free you to feel okay doing the bad thing that you might otherwise have unreflectively avoided.
3. Giving the sense that unethical behavior is pervasive. I've argued that people mostly aim for moral mediocrity. They aim, that is, not to be morally good by absolute standards but rather to be about as morally good as their peers, not especially better, not especially worse. If so, then changes in your perception of what is typical behavior among your peers can cause you to calibrate your own behavior up or down. If most people litter, or cheat, or selfishly screw over their co-workers, then it doesn't seem so bad if you do too, at least a little. (Conversely, if you learn that almost everyone around you is honorable and true, that can inspire you not to want to be the one schmuck.) Some ethics classes, perhaps especially business ethics classes, focus on case studies of grossly unethical behavior. This company did this bad thing, this other company did this other bad thing, still another company did this other horrible thing.... Without a complementary range of inspirational examples of morally laudable behavior by other companies, students might get the sense that the world is even fuller of malfeasance than they had previously thought, leading them to calibrate their sense of mediocrity down. If so many other people do so many bad things, then it hardly matters (perhaps even it's only fair) if I fudge a bit on my expense report.
Do ethics classes actually have any of these backfire effects? I think we really have no idea. The issue remains almost entirely unstudied in any rigorous, empirical way.