I tried searching for the origin of the phrase, but apparently nobody really knows.
The precise origin of the phrase ‘love someone to death’ is still unclear. That said, it is likely to have developed from the idea of a love that is so intense and overwhelming that it feels as though it will last right up until a person’s last breath.
The precise origin of the phrase, and the date of origin is yet to be determined, as is who first coined the term, and the date it became popularised and made common parlance.
I’ve never understood it to be referring to a particular person’s death, either the speaker’s or the listener’s. I thought it was just a meaningless intensifier, like “I love you to pieces”.
But you can do that simply with “I love you but …” without using the idiom. I don’t think there is anything special about “I love you to death” that invites a counterpoint. You can do this with many statements.