Book’s story is a mystery that likely will never get answered.
Mal: “You have to tell me that story sometime.”
Book: “No, I don’t.”
(paraphrased from Serenity) That was basically Whedon giving us his answer.
As for observations and guesses,
We know he was someone important to the Alliance. His identity and clearance got him a lot of respect from Alliance officers in “Safe”. At least at one time he was someone significant to them.
Question: would his status as “Shepherd” get that kind of respect? I don’t think that was what was intended by that scene. I get the impression that Whedon’s intent was that, if anything, Companions had more status than Shepherds.
He claimed to have gone off and spent a long time in isolation in the monastery. Now he could have been lying, but in “Serenity” (the original series premiere, not the movie) he is shown wandering around somewhat lost before finding Serenity. He would have had to have been really good at setting up his cover to be that in to it before he even knew he was near Simon, much less River. It is very doubtful he was a deliberate agent sent to track them. Furthermore, he never does anything to impede Serenity, Mal, or the Tams, and actively helps them evade and survive. So it is not likely he is a current agent of any means.
He did have a fair amount of knowledge your typical Shepherd wouldn’t be expected to have, from skill with weapons to dealing with criminals and cops to knowing how the Operative would work. All of which is just elaborating that he had a mysterious past prior to entering the monastery for that long period of reflection.
Was he an Operative? I’m not convinced, though it is possible that was what Whedon was hinting at by having the Operative in Serenity. Though I suppose every Operative would be a little different. He was certainly someone with more Alliance clout than the bounty hunter Jubal Early.
What are we to make of the one scene cited where he “thinks”, “I don’t care if you’re innocent or not. So where does that put you?” It does come of strangely at odds with his behavior, and malevolent in tone. It was rather more judgmental than a Shepherd should be. But he gave no outward appearance of that attitude, from dealing with River directly to doing anything to impede the group. Unlike Jayne, who several times took acts against the group or the Tams for his own self-interest.
It’s possible that Whedon had a couple ideas for Book’s past, and hadn’t decided what he wanted to do. Given Whedon’s style, though, I think he probably did have something concrete in mind as to who Book was in his former life and why he was on board. I mean, his role is to be Mal’s conscience and sounding board, but clearly Book has some aim for continuing to hang around.
I think whoever Book had been, he got disillusioned in his job and left for the monastery, but not on bad terms with the Alliance, just kinda quit and went his own way. After decades of reflection, he went back out into the world to explore and maybe find a way to atone, and thus stumbled upon Mal and the Tams. He decided they were a good starting place. His not caring whether River was guilty or innocent was meant in a more benign way, but River’s perception was off.
I don’t think he had a role in the Unification War, because he was in the monastery during that time. My guess is that he was either an Operative or some high official or military leader back when Mal and the gang were children, who took on a new identity “Derrial Book” once he joined the monastery as a sign of giving up his past life.