Watching TV is far less active than reading; everything is given to you. Reading requires both ‘decoding’ and a lot of active mental imaging. TV is a very passive activity, requiring almost no effort on the watcher’s part. Brain waves during TV-watching tend to be very low and passive, I understand.
In very small children, TV does little more than engage the ‘reptilian’ part of the brain near the brain stem. You might as well flash lights at them, almost. It is both so frenzied (esp. in much of children’s programming) and so passive that it frequently results in a weird state in which the child has a high pulse and adrenaline flowing but is also sitting very still–a sort of paralyzed ‘fight or flight’ reaction. Many people have observed that children often are whiny, fractious, or otherwise ill-tempered after watching TV and report that moods improve as TV-watching lessens. (I can also attest to this.) Many feel that it also tends to encourage a short attention span.
Something similar can be said of many adults. Some feel that they are wasting time, or even that TV is fueling depressive moods, but can’t seem to turn it off, either. It can be a big time-waster, taking the place of more active interests.
Some people seem to be vulnerable to an “addiction” to the TV, where they feel a strong need for it and get somewhat panicky at the thought of losing it. Children are also more likely to do this. While plenty of people can take or leave TV, others seem to be unable to wrench themselves away.
OTOH, TV can be entertaining, educational, and all sorts of things. There are tons of concepts that are far easier to learn with a few minutes of TV than with an entire book. As with most things, the key is moderation. TV is usually the mental equivalent of junk food–fun in small doses, but icky and unhealthy in large ones. I quite like TV myself, but only in small increments, and I usually have to have something else to do, or I feel useless.
FWIW, I do kind of wish that I had more interesting childhood memories than every episode of Scooby-Doo.
You may enjoy The plug-in drug, a rather opinionated book on the subject, or one of the many others out there. Endangered minds is another one and focuses more on children and education.