I know this seems like a silly question, but what does the ‘set’ in television set refer to?
I would assume it refers to the collection of devices which compose the TV (cathode ray tube, magnets, antenna, etc.)
It’s just one of the many meanings of “set” (which I believe is the word in the English language with the most meanings in the OED).
From www-m-w.com:
So perhaps the answer to your question is that “set” refers to the collection of components that make up the TV.
Then we have a computer set, a videocassette recorder set, a dvd player set and a refrigerator set, among other sets. 
Remember, TV was a direct descendant of radio. Hobbyists had radio (or “wireless”) “sets” that were a collection of distinct components: the battery(s) (often several jars of acid, not the small snap-in “dry cells” we all know today), the receiver, an antenna (another throwback monstrosity: often a clothesline-type affair), headphones, and – if you wanted to transmit too – a telegraph key. (Which brings up another point… I would not be surprised if radio borrowed the term “set” from the ye olde non-wireless telegraph, another arrangement that required many separate batteries and a key, at least.)
It’s no great leap that the lingo that described these early component radios stuck through the era of the all-in-one radio-in-a-box to the age of the TV-in-a-box.
The real question, IMO, is whether “set” was introduced by early radio (or telegraph) practicioners to describe a collection of components (as in, “I have a new set of batteries”) or as a shorthand of the term “setup.”
And to add just a little more, I remember from my high school French class that the French call a radio “poste de radio” and a TV “poste de television,” which I guess roughly translates to “radio set” and “television set.”
We forget these things were named a long time ago and terminological fashion changes.
As someone already pointed out, “TV set” was probably a derivitive of “radio set.” “Radio set” was coined somewhere around 1895 when Marconi built the first experimental transmitter & receiver that actually worked.
Read any book on any topic written around then and you’ll find a lot of language usage that sounds weird to our 21st century ears. Not undecipherable, just a little off. Someone from back then would be just as mystified at our language today.
Way back whenever it was that TV was just starting out, the TV set truly was a set of boxes. The tuner section in the earliest sets was a converted radio receiver, adapted to handle video as well as sound. The picture tube and its high-voltage power lived in another box.