I suppose you can call it an adverb. It’s part of a phrasal verb, which to my knowledge is generally treated as a single unit, even if it takes the form of two words. That’s the problem with such rules, though - they result from only the shallowest sort of acquaintance with grammar - since we’re taught that “to” is a preposition, and we’re taught never to strand a preposition, people develop the notion that “take your hat off” is somehow improper, even though that notion is wrong on both counts: grammatical English has always stranded prepositions, and “off” isn’t a preposition here anyway.
Thus is the sorry state of knowledge regarding linguistics in this country.