Heh heh, he said headlights.
Evil Captor, that is a great description of the process.
When I was writing my novel, I had something absolutely huge come out of nowhere. I tried to fight what was going on with my character and “reign him in”, but it flat-out did not work. Finally, I let him lead me in the direction he wanted to go, and it worked out so much better.
From what my readers have told me, that was the point at which the character came alive to them.
I remember thinking at the time that it was a very powerful, magical, and mystical moment.
I’m kind of halfway between an outliner and a pantser. I’ve never had a novel published, but I’ve written five of them that are published on my website (and that had a decent size fan following when I was posting.) In the first one, my villain was somewhat one-dimensional–he was an antagonist for the heroes, and had a few vague reasons for his hatred of one of them. However, after I finished writing that one and figured I was done, he started to talk to me (as did the hero he hated), and explained to me that there was a lot more to their story than I was putting down, and I owed it to them to explain the whole thing. This led to four more novels–during the course of which the villain became a lot more interesting, developed an in-depth personality and backstory, and ultimately ended up making a sacrifice that saved not only the hero but arguably the world (this series was set in the RPG Shadowrun universe, so it had a definite sci-fi/fantasy/cyberpunk vibe).
I had no idea any of this was going to happen when I finished the first novel. Hell, I didn’t know there would be more than one! But each one was fairly rigorously outlined, at least for the main plotlines. I let the characters interact the way they wanted to, though, since my stuff is heavily character based. They rarely let me down, though I did have to rein them in occasionally when they tried to wander off the plotline. 
I’m not a writer, but I can tell you from a reader’s perspective that is is jarring when the characters don’t do what they’re supposed to do/what they’ve been written to do. I’ve read a couple of novels lately that had great ideas that were poorly executed, and I kept running into that - the authors kept forcing their characters to do things (or things just kept happening for no particular reason), and it made for an irritating read. I’ve been a huge fan and supporter of Stephen King for a long time for just the reasons he mentions - his characters go where they’re supposed to go, whether I like it or not.
Same.
I know where I’m going to start, and I have ideas about where I want to end up, but don’t know how I’m going to get there until I’m actually writing - and sometimes the path from point a to point b decides to hit point c instead.
And even when I do get from a to the intended b, some of the details of the trip surprise me.
Two examples from something I’m working on now (started as a novel, now I’m reworking it into a comic)…
I have 3 main characters, 2 male, one female. I was writing some incidental dialogue between them and found one of the male characters expressing an attraction to one of the male background characters.
Huhn.
I hadn’t intended him to be gay, and didn’t realize that he was going there until I was in the process of writing the line. But once that line spilled out…hey, it worked.
I’ve since discarded the scene, because a lot of the surrounding stuff was awkward, and there was no place to put the scene in the revised version…but he’s still gay.
The other two, I had no intention of having them hook up, but as the story progressed…sure enough…chemistry! They were actually positively adorable. I ended up creating a new character to pair the other fellow with just to not avoid having the straight characters pair off and leaving the gay guy single. (They’ve been doing what I want in that respect, at least.)
I’ve written short stories with a vague outline and was once surprised when one of my characters reared up and hijacked a story to a whole different level. I rewrote it furiously, and it was like riding on a runaway horse (she said, opening a new box of Little Debbies to keep her energy up). Quite exciting in a way, like discovering your fat middle aged neighbor next door once climbed Mt. Everest.
Ultimately it all comes from inspiration. Some authors have the whole inspiration ahead of time. Some authors get inspired during the writing process.
I’m a tech writer, and I’ve only written a few short stories. I wondered about that surprise thing, too. This was my experience: I wrote a murder mystery short story. At the beginning this guy goes to his office late at night because he sees a light on where there shouldn’t be one. When he enters the lobby, he sees a post-it note on his door which is on a second-floor mezzanine. He goes upstairs, takes the post-it off the door and reads it. As the writer, I didn’t know what was on the post-it. I hadn’t gotten that far.
The murderer comes up behind the guy and pushes him down the stairs and kills him. The next morning, his colleagues come in to the building and find his body. One of them takes the note out of his hand, reads it and puts it in his own pocket.
I knew who the murderer was from the beginning and went about telling the story, introducing red herrings, etc. At the very end of the story, it occurred to me what it had to say on the post-it. It just fit very neatly and tied everything up. It was indeed a surprise to me and a truly delightful feeling to be surprised that way.
Another time, and this isn’t about writing, but about an art project, I was taking a class and our assignment was to do a series of three drawings around a theme. We were to take an object and present it in three different settings and tie the three together. I was quite stumped. The class was in summer school and I only had five days for the project. I just started playing around with stuff, all in a sweat, because I didn’t know how I would finish the project by the deadline. Finally I somehow cranked out the three drawings and they were QUITE good, if I do say so myself. And when they were mounted and on the wall for critique, lo and behold, each one was a window. You might not believe this, but I assure you, I did NOT set out to do three drawings of windows. I grappled with each one separately and somehow they all turned out to explore the theme of a window. That’s why they call it art, I guess. I’m still pretty astounded when I look at those three drawings.
Yeah, writing is as much about finding out what will happen as putting down the stuff you already intend to happen. I don’t think I’d bother to write at all if I knew every twist and turn ahead of time.
A friend of mine has been working on a novel in her spare time for a while. She hasn’t told me much about what it’s about, but she did mention at one point that she’d realized she didn’t know how it was going to end anymore. She’d decided on an ending pretty early on, but after she’d written a big chunk of the story she thought about her intended ending and realized “No, my heroine wouldn’t do that! She’s just not the kind of person who’d make that decision.”
I haven’t written much fiction myself, but I did once write a story where I was surprised by a plot development after the story was finished! I took a creative writing class in college and wrote a sort of ghost story that was well received by my classmates and that I successfully submitted to the school’s annual literary magazine. So a lot of people I knew wound up reading this story and telling me what they thought of it. I’d wanted to experiment with an unreliable narrator, and had my narrator experience what were either psychic visions or hallucinations about a murder that had happened many years before. I also deliberately wrote the story so it wasn’t clear if the narrator was correct in his belief that a ghost or demon had influenced this murder.
Every single person who read this story and talked to me about the ending said something like “It was cool how it’s not quite clear whether the narrator was crazy or if he was possessed by the ghost” or even “It was great how at the end you realize the narrator has become possessed by the ghost.” I had not in fact intended to suggest that the narrator was possessed by the ghost, and as best as I can remember such a thought never even crossed my mind. If anything I meant to hint that another character might have been influenced by the ghost. But not only is there nothing in the text of my story to contradict the possessed-narrator interpretation, it provides a reason why my narrator started having visions of things that happened before he was born – something not otherwise explained, unless he was just crazy. I think it also makes for a better and creepier ending if one suspects that a narrator who’s been worried about ghosts has unwittingly fallen under the control of one. Looking at the story long after the fact, this seems like such an obvious and satisfying interpretation that I can hardly believe I didn’t intend the reader to see it that way, but I didn’t.
I remember this experience whenever I hear someone claim that whatever interpretation the author intended is the only “correct” one!
He actually goes a bit into this process in the final scene of Misery where a writer-blocked, post-Annie, Paul sees something on the way back from his publisher, thinks about it, and then sits down and starts to write the story, tears coming from his eyes.