First let me say, I love this thread. No really, I’ve spent the better part of the last decade studying, researching and teaching classes in various aspects of leisure and how people spend their free time. Yep, that’s right–I get paid to work at a job where I study fun. Ain’t life grand? 
Anyhoo… I’ll offer a couple different answers.
First, there is indeed a general sense (right or wrong) that watching television is a lower form of recreation. Why might this be? Well, some would argue because it is nothing more than a passive exposure to various stimuli and not at all a true engagement (where one is actively involved in something). We can recognize that there are certainly degrees of involvement in watching television: being engrossed in a movie or newscast to slouching slug-like on a couch mindlessly watching the images flicker on screen. Thus, generalizations on television watching are subject to these criticisms. It might also be said that television watching is a lower form because it promotes isolation. Interestingly the same criticism was leveled against reading novels (as opposed to newspapers). In the mid 18th Century, when novels first gained prominance, some folks were worried that this type of leisure would insulate people from their communities and give rise to a nation of isolated individuals who have no concerns for their fellow humans. Fast forward 250 years and the same critique is leveled at television.
Another take on the more general question about why ‘escapist’ behavior is considered sinful can be traced back to Ancient Greece. The ancient Greek word for leisure, schole, means "serious activity without the pressure of necessity. To Aristotle, in particular, the highest form of schole was to contemplate the good and noble truths of life (whatever those are). Spending one’s days this way was a virtuous pursuit, while spending them otherwise was at best neutral and at worse a vice. All day in front of a television would be a vice (read: sinful) according to Aristotle for two reasons.
One, it offers no advancement towards understanding the good and noble truths of life. Reason played a preiminant role–above simple pleasure-seeking. This idea echoed in John Stuart Mill’s critique of Jeremy Bentham’s ‘Calculus of Felicity’. Mill worried that leisure choices are not simply quantitative but are also qualitative. As such, some choices are in fact better, more noble, than others. Mill probably wouldn’t argue that pursuing ‘lower’ or ‘baser’ leisure pursuits is sinful or even unethical, but as a founder of a teleological ethical system, he probably would encourage people to pursue those things are more good rather than less good.
Aristotle’s second critique would again fall back on the notion that anything that separates one from his/her community is bad. Living a good and worthy life required participation in the polis.
One question that continually creeps up in this type of discussion is “Why do we leisure at all?” Is it so that we can refresh ourselves for work? Or do we work only so that we can afford to do those things we aspire to in our free time?