In a libertarian society...

That’s where we differ. I (and quite a few other people, including the founders of the United States) think that rights are non-negotiable. As in “inalienable”. No one can take away the man’s rights except the man himself, by his own decision.