I don’t disagree. The right to privacy probably doesn’t help all that much.
Not federal law, no. Although, without checking, I’d wager that federal law includes some similar prohibitions for land under federal jurisdiction. Ah, what the heck, I peeked. http://www.law.cornell.edu/uscode/html/uscode18/usc_sec_18_00001384----000-.html (strangely located adjacent to the posse comitatus statute). Also, there’s the Mann Act:
So to the extent that swinging is illegal in a state, taking somebody to a club (especially if the club is across state lines) could be a federal crime.
And the Phoenix Ordinance (pdf) was drafted with swing clubs in mind.
Generally, though, you are correct. Most US swing clubs exist with little fear of persecution.
Best example of that is section 287 of the Criminal Code, which criminalised abortion without the permission of a therapeutic abortion committee. It was struck down in 1988 by the Supreme Court, leaving abortion completely de-criminalized in Canada, but the offence is still in the Criminal Code.
Heck, they even carried it forward into the new Revised Statutes which came out in 1988 after the Morgentaler decision, re-numbered from s. 252, the version that was struck down.