In this usage, “to death” means “extremely.” Originally hyperbole in more understandable phrases like “he’s going to work us to death” and “you scared me to death,” it’s been appropriated by phrases where the feeling (something like “as much as I can stand”) is the same even if the literal meaning is way off.