The people you are wondering about are most likely coming from one of two groups.
The first group is young, confused, and seeking for something to latch onto as a way to form an identity. They could just as easily have become football fans who turn their noses up at other sports. The appeal of anime/manga for these kids is twofold. For one, it gives them an opportunity to feel superior, because after all, anime is “mature” and American TV / movies (and by extension, fans of American TV / Movies) are therefore inherently inferior. Secondly, Japan is “exotic,” but it’s an easily-accessible and clean exotic, without any unpleasant aftertaste or consequences. So, you get the best of both worlds; you can be strange and obscure and be a unique butterfly that is “diversified” and “cultured” without actually having to make the effort to learn any dirty details and have a true understanding. Sort of the same way that people who want to travel to other countries to learn about other cultures usually will just eat in a couple of restaurants with other Americans.
(Not that I can be overly-critical. We’re all guilty of that kind of attitude to some extent. Most of us would say that we want to save the environment, but we would hardly consider giving up our air conditioning or other energy-hogging luxuries.)
The other group is made up of single video-gamers who are into Asians. Since Japan is the origin of most video games, the logic seems to go like this: get really into Japanese culture via easily-accessible imports, learn basic Japanese, go to Japan to teach English for a few months, meet some cute Japanese chick/dude, and score the perfect wife/husband who won’t have any problem if you want to spend the entire day sitting on the couch and playing games. This group tends to give up somewhere around step 2, because learning another language is, like, hard, and so they end up just being anime fans instead.
Sure, sure, there’s lots of other people who like anime for many other reasons, and I’m sure there’s lots of white folks who totally “get” Japanese culture and all that other crap. But the two groups listed above are very large, and very obnoxious, and they’re probably the ones that got on your nerves and led to your question. In both cases, the anime itself isn’t the attraction - it’s simply a means to an end, though not necessarily an unpleasant one.
On a related note, my wife is part Japanese, and neither one of us has any strong feelings one way or another for anime. She does, however, get really pissed off whenever a new nerdy acquaintance asks her about it. The worst ones try to speak to her in Japanese and she has to respond with, “What the fuck country do you think you’re in?”