I disagree. To say your point another way, killing an animal is never justified except life or death, by which I assume you mean said animal is attacking me and I must kill it to avoid my own death.
You are correct that, unlike times past, choosing to eat an animal is no longer a matter of life and death, nor using an animal’s skin to stay alive against the cold. We have non-animal alternatives to both food and clothing. I would have to be convinced that these alternatives are sufficiently available to everyone. But even if if we’re not there yet, it’s conceivable we could get there if we tried hard enough. So for the sake of argument only I could stipulate that everyone in the West could choose to be a vegetarian.
It’s a stretch, though, to say killing an animal for fun is the moral equivalent to choosing to eat an animal to survive (although I’m a teensy bit more sympathetic to the equivalence of wearing an animal). Clearly there is a different mind-state, a definite moral deficiency in someone who would actively commit the killing of an animal solely for the entertainment value provided by the killing act itself.
Someone who eats meat is responding to a natural hunger impulse with a food source we’ve evolved to be able to process. To take the argument to the opposite extreme, one could argue it is immoral - or at least inhuman - to suggest as a society we ban meat as a food source.