I'm going home. Why no "to"?

For any other location, you would need a “to” there. Why is “home” exempt?

Not ANY other - “I’m going downtown”. Or, for that matter, “I’m going out”.

To is for specific places. “the movie theatre”, “my house”, “Fred’s room”. Home is a general concept like “outside” and “downtown”. Note that you would say “I’m goign to my home.”

I’m going north.
I’m going underground.
I’m going in the backdoor.
I’m going crazy.
I’m going to a home.

Traditional grammarian explanation is that they’re nouns (and adjectives) used adverbially – or possibly as “predicate adjectives,” as in Nick’s example of “I’m going crazy,” which is not “I’m going crazily.”

Better explanation is that it’s a survival of the pre-Old English locative, a noun form used without preposition to denote location. (It had some case ending, I think Dative, in Old English, being the survivor of a proto-Germanic or Proto-Indo-European locative case.) It’s sort of a fossil form. “I’m going home” or “I’m going downtown” is proper, but not “I’m going library” or “I’m going Florida for vacation.”