I was a TA for a HUGE freshman History 101 course. There were so many in the class we couldn’t take attendance. When it came down to the final grades the prof would give people a boost upward if I ever saw them in my office. The thinking being if they stopped by to talk to me, they were giving the class some effort (I made up the tests and was always willing to discuss what should be studied and what could be avoided).
Students? Sure. It’s other faculty members who plop their asses down and want to chat that waste my time…
I have a colleague who does this, albeit infrequently. Even when I turn my back to her and continue to type, file, etc. she just stays there. 
Oh, gods yes. We’ve got a “Camping Carl” on my floor, who will literally go from office to office to talk (in his outside voice) about NOTHING. One person boots him out, he just goes to the next office. He’s been reprimanded about it, to no avail.
Well, since I’m a half-administrator it’s part of my job to be available for the “Camping Carls” to come by and bitch about anything and everything. I’ve since learned two things:
- It is true. The emotions are high because the stakes are so low.
- If I want to get work done, I need to just shut the damn door.
When I was a TA, I wasn’t particularly interested in having students show up during office hours. I’d rather read a book or surf through Usenet. (Note: I was not a very diligent grad student…)
Just to pile on: yes, absolutely. Office hours conversations with students let teachers know what direction the term is going, and are probably the best thing that an instructor can do to help students succeed. Never hesitate.
Unless it gets excessive, of course.