I only have two sections, fiction and non-fiction. The fiction is alphabetical by author’s last name. Within the same author, titles are organized in publication date order (if it’s a series) or alphabetically by title.
I tried to come up with a good non-fiction system, but I gave up and eventually just went with the author’s last name.
Physical books, as opposed to ebooks, get sorted first by format. Most of my collection is mass market paperback, and I adjust my shelves to that height. I also have trade paperbacks in various sizes and hardbacks. Within the format, I sort by fiction genre or category and then by author. That is, my science fiction/fantasy collection covers several walls, and then I sort by author if it’s a novel or a single author anthology. In cases where the anthology contains stories by several authors, but it’s based in one author’s universe, then the anthologies get filed under that author’s name, even if he has no stories in that particular collection. For instance, there’s a series of stories called the Man-Kzin Wars, set in Larry Niven’s Known Space universe. Those books get filed under N for Niven. Similarly, there have been stories about Nero Wolfe that have been written by someone other than Rex Stout, but those books are filed in Mystery under S for Stout. I remember that they are ABOUT Wolfe (and Archie), and so I look for them in the Stout section. Hey, it’s what works for me.
I file the trade paperbacks and hardbacks together using the same guidelines. I try to use logic when determining which shelves hold which category. For instance, the largest group of shelves is dedicated to SF/fantasy. And I do file my SF and fantasy together, as sometimes the line gets blurred.
I organize books on shelves professionally. At home, the kids have no shoes.
I don’t organize them the way I’d like to, because six-dimensional bookshelves are a bit hard to come by (I think the Ikea in R’lyeh has them, but that’s such a commute).
What I actually do, then, is a shelving unit for each genre, then all of a given author together, then within each author each series is together. I try to put thematically-similar authors together, but by this point I’m generally scrambling just to find ways to keep authors from overflowing their shelves, and filling in what spaces I have. Paperbacks and hardcovers are all mixed in together, unless a book is just plain too big to fit in that way.
One bookcase contains SF, another is for fiction. Both these are arranged alphabetically by author’s surname.
Another is for cookbooks, gardening books and craft. No specific arrangement within each category.
I have various other bookshelves scattered throughout the house, one of which contains my photo albums, photographic books/magazines and computer books. Another is for dictionaries and books on language. I have two others which are non-fiction bookshelves, arranged haphazardly by type (history, science, religion).
Cookbooks out on the porch on a dedicated shelving unit.
Nonfiction (which tend to be the tallest) on one unit more or less by topic, one sort of blending into the next:
Anatomy/Phys/Biology>
Nursing>
Historical Medicine Books (turn of the century “Marriage Manuals” go here)>
East-West Medicine Crossovers>
Traditional Chinese Medicine/Acupuncture>
Nutrition>
Western Herbals>
Children’s Herbals>
Pregnancy books written for expectant parents (the Maternal Fetal Medicine books are with the Nursing books, after much angst and rerereconsidering)>
Parenting>
Child Psych/Development>
Psych/Abnormal Psych
The other bookshelves house Fiction, alphabetical by author and, within author, in series order (generally chronology within the story, although a few in publication order instead - Narnia gives my OCD fits). If an author has more than one series, my favorite series goes to the left, lesser favorites to the right.
The bottom shelves of the Fiction units have yearbooks, programs from shows and other misc. stuff on one side, and Occult texts, eagerly started and quickly abandoned journals and my Book of Shadows on the other.
I lost, literally, thousands of books in my divorce. Saddest part of the whole thing. 
Alphabetical by author, then chronologically within the story lines the author is writing. So, the Valdemar novels start with Magic’s Price, then the new series with Mags, then the Oathbreakers, etc., regardless of when they were published.
Similarly, in my DVDs, I have Hulk, Iron Man, Iron Man 2, Thor, Captain America and Avengers between Avatar and Batman, because they’re all Avengers movies.
So:
Eight inches
Eleven inches
Five inches
Four inches
Nine inches
One foot
One inch
Seven inches
Six inches
Ten inches
Three inches
Two feet
Two inches
(Or do you go metric and arrange by centimetre?)
By topic, though that gets rather fine-grained for collections such as “Chicago,” which is some 20 shelves.
I do use one unusual technique for guidebooks: they’re in rough ZIP code order. I know the three-digit ZIP codes for most American cities, and it’s a way to make a two-dimensional array into a one-dimensional arrangement. I’m not obsessive about it (I won’t bother to actually look up a ZIP code), and the foreign ones are arranged from Asia through Europe to the Americas. So maybe it’s more accurate to say I arrange in time zone order.
I do business cards the same way, because I can always remember about where someone’s office is, but never their name.
I’ve got 15 bookcases with a total of 64 shelves.
They’re organized by subject. The main categories:
Reference
Birds
Mammals
Reptiles and Amphibians
Fish
Insects
Other invertebrates
Other natural history
Paleontology
Other science
US History
Panama
Other Latin America
Africa
Asia
Oceania
Travel
Art
Cookbooks
Baseball
Humor
Fiction
Plus a bunch of minor topics.
Grouped by subtopics or regions within topics, with occasional deviations for odd-sized books that will only fit on the larger shelves.
I’m alphabetical by author, with “series” books in sequence, followed by any “stand alones” in any order, without regard to title or pub. date.
Loosely by topic, and my favorites on the bookshelf in my bedroom.
Okay…so for all y’all who do alphabetical by author’s (I assume last) name…what do you do with the Marion Zimmer Bradleys and Orson Scott Cards of the literary world? Dang three named authors…
“B” is for Bradley, “C” is for Card…bot so hard, IMO.
That’s what I do, actually, but then I stare at it for long minutes, wondering if it’s really Bradley, Marion Zimmer or if it should be Zimmer Bradley, Marion.
Then I give up and go look for the tequila.
I sort haphazardly with a vague attempt to group by subject. Generally speaking the main categories are Rock n Roll/Entertainment, Travel/Natural History, Reference and random fiction.
Obviously, it depends on if the author is American or not.
The first Orson Scott Card book I read was borrowed from the library. I found it in the ‘C’ section, so that’s how my own bookshelving system works.
Pretty much this. Most of my books are science-fiction/fantasy, but there’s a fair bit of non-fiction (history, professional references, science, biography, etc.) mixed in, usually alphabetically by topic. For example, the books I have on Richard III are in ‘P’ for Plantagenet. Although I think And the Band Played On may be in the S section, for Shilts …
hmmmm.
(I may have to start culling. I’m running out of room!)