Professors get nothing for most custom textbooks. We just work with the publisher to choose specific content, often from a database. I use a custom book each semester, so our bookstore buys them used. They tend to be cheaper anyway, since I choose only the content I need.
At my (state) school, we cannot profit from textbooks.