What's the root of the opprobrium against desiring pleasure?

I suppose a yogi might say something to the effect that there is nothing bad about seeking pleasure and enjoying life, but that such behavior will use up energy that you could otherwise spend meditating, and distract you from your personal spiritual development. It could be a personal thing, not necessarily some macro societal consideration.