It comes from the pulp magazines of the 1930s. The pulps were lurid, mostly subliterary, and aimed at the great unwashed. Writers ground out stories like sausages, with even the best writers earning possibly two cents a word and most of them a half-cent a word. To make the minimal living wage you pumped out tens of thousands of words of week. Lester Dent did a 60,000 word novel every week for years. Guess how literary they were.
The pulps were printed on the cheapest pulp paper and had the loudest and most eye-catching melodramatic covers, often of scantily-clad women having evil things to them by fiends. And those were just the westerns! [rimshot]
For decades, when anyone read sf they read it in the pulps. Amazing Stories started the modern separate genre of sf in 1926, followed by many other titles, most notably Astounding Stories of Super-Science in January 1930. How could anyone possibly respect fiction being published in Astounding Stories of Super-Science? Yes, the title was later shortened to Astounding Science Fiction and John W. Campbell took over as editor in 1936 and he published the early work of Asimov, Heinlein, Del Rey, and L. Ron Hubbard. But those stories had next to none of the literary value of even the average story in the Saturday Evening Post, let alone the American literary renaissance that happened simultaneously in the 1920s with the rise of Hemingway, Fitzgerald, Faulkner, and so many more.
Every time sf tried to demand some literary respectability something would happen. Digest-sized magazines like F&SF and *Galaxy * appeared in 1949 and 1950 and their editors demanded better written product than the bilge that Campbell often printed but that coincided with the advent of sf on television - kiddie shows like Tom Corbett, Space Cadet - and the 50s horror/monster tales that were Z-grade movies.
SF tried to become respectable again in the 1960s but then Star Wars and the Star Trek movies diverted the audience to gosh-wow adventures and even the print market became half tie-ins to media sources.
Go into any chain store today and look at the f&sf racks. Fantasy books have a person in a medieval uniform holding a sword. Science fiction books have a person in a futuristic uniform holding a blaster. If you can judge a book by its cover - and with these books you can - you know exactly what you’re getting and its not literature.
In the UK, by comparison, the pulps were never as dominant and sf was never so closely identified with the pulps. Writing sf in the UK has always been far more respectable than writing sf in the US. And its still looked down upon as second rate.
We would have killed to be thought of as merely second rate, but in fact the field started out as tenth-rate and had to claw its way up a few rungs of the ladder to whatever precarious position it holds today.
I long ago gave up any hope of sf becoming respectable, even though you can open any issue of the New York Times Book Review and find a novel that sounds suspiciously like something from the field. It’s not marketed as f&sf, though. That’s the kiss of death.
SF is looked down upon as something less than literature because of things that happened before you were born. It never overcame that taint and the most popular writing in the field today doesn’t even attempt to. So why should anything change?