Why the word 'Skinny' in skinny dipping?

I was watching How I Met Your Mother last night and one of the characters suggested that they go “skinny diving” (Sky Diving sans clothing.) This got me to wondering: why do we call it *Skinny *Dipping?

According to The Maven:

Incidentally, I usually find that googling “word origin” or “phrase origin”, together with the phrase I’m curious about, quickly yields the answer to these kinds of questions.

The derivation I’d always assumed simply from the nature and the use of the term by generations of kids is that one goes for a dip “in one’s skin” (as opposed to with swimwear on). Parallel to it might be the two pickup teams competing in intramural athletics where one retains their T-shirts and the other removes them, becoming respectively the Shirts and the Skins.