By language, so my mother can find the stuff in Spanish easily; then by subject and/or author but grouping things as much as possible. For example, all my Pratchetts or all my Pérez-Reverte’s are together, but being different languages they aren’t even in the same bookshelf (the P-Rs are in the living room, the Pratchetts in my bedroom). Series are in chronological order.
In my Nook.
Okay, when I used to keep a paper book collection, I kept them organized by topic per bookshelf (fiction bookshelf, non-fiction bookshelf, fantasy, sci-fi, etc) and then alphabetically by author from there, with series grouped together.
My non-fiction is divided into biographies, poetry/plays/quotations and general non-fiction (I’ve considered further subdividing it, but haven’t gotten around to it yet). Within these categories, it’s arranged alphabetically by author’s last name. Even for biographies - the people who wrote those books deserve to be acknowledged, damn it!
My fiction is divided into historical fiction, SF and other. The historical fiction is arranged chronologically, even when this means breaking up a series. SF is arranged alphabetically by author’s last name. Books by the same author are in publication order, unless they’re part of a series where chronological order is not the same as publication order. In that case they’re in chronological order. Other fiction is alphabetical by author’s last name.
I tend toward categories first, then it varies. In some cases, categorizing gets overridden by shelf/book sizes. 
By general category, then by height, then by series/author. If the current shelves aren’t in the best condition, then by weight as well.
After that, by purchase date and distance to bed.
I generally try to keep series together, and in the right order, but beyond that, I just shove them wherever they’ll fit. (‘By series’ is not entirely compatible with ‘by author’, since I read a number of series that have multiple authors.)
I’m a librarian, and other than using LibraryThing (as mentioned upthread) so I know that I’ve got/read something, I pretty much have zero organization. 
Categories, mostly - the Disney books are all in one bookcase, non fiction in another couple of cases, then fiction in a few more. The Zombie/horror stuff all goes together tho, fiction & nonfiction (King has a shelf to himself, of course)
Fiction - by genre, then by author - not alphabetical, but rather thematically -so Spider Robinson & Heinlein share a shelf, as do Harlan Ellison & Kurt Vonnegut.
NonFiction - by genre, then author, mostly. The mass media case is broken down into separate shelves for movies & TV. I try to keep science, sociology & history grouped separately, but some books span all 3 - (Mary Roach & Mark Kurlansky, I’m looking at you)
I do this, my friends think it’s hilarious – organising by colour, that is.
I’m a graphic designer, I can’t help myself. It was a right pain to do, mind.
I’ve seen some used bookstores that file their horror and mystery books together. I call it the “books with black covers” section.
Whatever the Library of Congress does.
I toyed with the idea of LCing my whole collection, but that wouldn’t be practical. Most notably, doing so would my Hmong books all over–some would be in Asian history, some in history of American minorities, some in world folklore, and at least one would be in medicine. As they sit, they clump as I need them. If I’m working on an East African project, all those books are together. If I’m working on a Mexican project, all those books are together.
I’m not real consistent with subject matter, btw–there’s a lot of truthiness and aesthetics involved. I “should” have my Russian folklore books in with Folklore, but I needed to fill out my Russian shelf, so they went there, as did a couple of Russian Novels. Chinua Achebe goes in Africa instead of Literature because I use his work more as a source for African history and culture than as great fiction.
“Scatter.” The missing verb is “scatter.”
Years ago I brought my grandmother to my favorite local used bookstore, and she was horrified to find that they didn’t separate mysteries from general fiction. At that time I wasn’t reading mysteries, so I had never noticed.
My biggest issue with the store now is their treatment of trade paperbacks. Their hardcovers and mass market paperbacks are neatly sorted and shelved, but their growing collection of trade paperbacks are stacked sideways in a back corner, in no particular order. I suppose that they don’t want to shelve them in the hardcover room, but they don’t fit on shelves built for mass market paperbacks. They’re priced like hardcovers: around $4 each, instead of half the cover price.
It’s how my grandma organized, but with the caveat that children’s books had to be on the lower shelves. Thus, the Just Williams left behind by her children were directly under great-great-grandfather’s “do not dare touch it” Universal History, because both had red backs.
The Universal History is now in my entranceway. The Williams went to one of my uncles and I’d bet my kitchen that I know which cousin has called dibs on them.
Fiction: by author, internally by series and series order if applicable.
Non-fiction: by subject and author.
Odd or oversized books: on a shelf that fits them, generally grouped by some kind of criterion if possible.
I’ve only got about 600 books at home right now on three bookshelves. There’s a small overflow / trade in / new purchases shelf too. I’ve been living in Japan for several years, and while I’ve built up a collection over time, I’ve intentionally culled aggressively and limited my purchases. I’ve got another 1200–1500 or so in storage in the US. Those will need another several bookshelves to fit if/when I move back.
the shelves are organized by subject, also I have an electronic catalog of my collection. I made it with All My Books program. I am tracking the books I’ve read there too.