What, in your opinion, is the best science fiction book (novella, short story, etc.) that failed to win either award? Clearly, we are talking published since 1955, preferably since '65, so it had a chance to win a Nebula.
I have to think that Titan by John Varley ranks high up there.
Quite frankly I think that the Hugo and Nebula reflect popular sentiment of the times well but typically don’t coincide with the “best” SF novel of that year or even the most memorable. You might as well ask “What’s the best SF or Fantasy novel?” I’m sure if someone ran through their top three books they’d have something in there that did not win. The question is just way too broad.
The following books have all made the final Nebula short-list for the ballot:
Slaughterhouse-Five, by Kurt Vonnegut Jr. Gravity’s Rainbow, by Thomas Pynchon Invisible Cities, by Italo Calvino Ragtime, by E. L. Doctorow Ridley Walker, by Russell Hoban The Handmaid’s Tale, by Margaret Atwood Mary Reilly, by Valerie Martin Cloud Atlas, David Mitchell
It’s obvious to one and all that no book that is by anyone out of the core group of sf writers will ever win an award. Those who nominate books in SFWA have all but given up even bothering to look for good books outside the “ghetto.”
The Hugos traditionally never ever nominated such a book. There’s been a slight shift recently because five of the last six winners have been either British or Canadian. Three are certainly well known in the field: the Canadians Robert Sawyer and Robert Charles Wilson, and the ex-pat Britisher Neil Gaiman.
The other two are J. R. Rowling, who won for the first Harry Potter book, and Susanna Clarke, who won for Jonathan Strange & Mr Norrell. Rowling isn’t part of the field, but the Hugo is a fan award and the overwhelming acceptance of Potter in the fan community drove the award. Was it the best book of the year? Doubtful. Was it better than the other nominees? Never read 'em.
So what about Clarke, who is a full-fledged mainstream sensation? She has much closer ties to the field than the mainstream press would have you realize. Wiki says:
And who does she live with? That same Colin Greenfield, a prominent British writer and critic of sf.
So there’s really no point in talking about best books to fail to win if you’re going to limit the conversation to core sf. Dozens if not hundreds of great mainstream with sf themes and content, books usually called “slipstream,” including those above, have been published over the 40 year lifetime of the Nebulas. I guarantee that they will continue to - at best - make the low end of the ballot and never win. That’s not to say that some of the books they would be up against could not forthrightly win on their merit. But the point is that they will never have to.