I think in order for it to be sexist there has to be a difference in what the sexes are allowed to do, and I think that depends upon how it’s phrased. If one were to say that, without SSM, men can marry women but not men, but women are just the opposite, then it could be said to be sexist. If, however, one states it that a person can marry a member of the opposite sex, but not the same, then it’s not sexist. In my mind, it’s sort of like saying that it’s sexist to have different restrooms. One could say it is because a man can use the men’s room but not the women’s room and women just the opposite, or one could say it’s not because each can use their own room but not the opposite. But that’s sort of just a semantic argument, so I doubt that really means much
While sex is a determining factor, it’s not the sex of the individual, but the sex of the couple that makes the determination. It is not that a man or a woman cannot marry, but specifically the sex of both individuals, and thus depends on the relationship of the sexes of the couple. As such, it seems to me to be misrepresenting the discrimination, since it necessarily requires information about the relationship and not just the sex of one individual.
I do think that opponents of inter-racial marriage are racist, and while that seems analagous on the surface, I don’t think it is, and I think what I see as the difference might help illustrate the point. That is, those opponents don’t have an issue with two people of the same race marrying, but specifically “one of us” marrying “one of them”, there’s no problem with “one of us” marrying “one of us” or “one of them” marrying “one of them”. However, if the SSM discussion, the “one of us” is a straight person and “one of them” is a gay person, and so the opponents aren’t necessarily against “one of us” marrying “one of them”, as a gay man can marry a straight woman or a lesbian can marry a straight man just fine; hell, they’re not even necessarily against “one of them” marrying “one of them”, as a gay man could marry a lesbian. The entire issue is about “them”, and “them” is not defined by sex, but by sexual orientation.
But really, I’m not even sure what saying it’s sexist might add to the discussion. It may even be sexist in practice, but that’s almost certainly not the intent of most opponents of SSM. I think it distracts from the crux of the issue.