Legalities and ethics aside would this work on practical biological level?
Anecdotal evidence only to follow:
I used to know a guy that was a baddy. I mean, a really, really, really bad baddy.
He got a brain tumour (karma is a bitch. Yes, I know that’s not how karma works).
He needed to take estrogen as a treatment.
He got all funny and weepy and wanted to talk about his feelings.
I wish I was kidding - this big, bad, bearded, hell’s angel, type shit disturber and he was sitting there weeping into his hands and talking about how sensitive he felt about everything.
Personally, I say it can’t hurt in a prison situation.
Violence is a learned response as well as an instinctive/hormonal one, so at best you’d only affect some violent felons. And the ones who weren’t affected would have every reason to be even more violent in revenge, given how screwed up doing this would make their bodies.
Alan Turing estrogen treatment
-Not a happy story.
Only for three weeks out of four… 
Appropriate chemicals would turn “violent felons” docile. Although the mechanics of how to be violent may be learned, the predisposition itself needs genes and their assorted phenotypic chemical expressions in order for that learning to be converted into behaviour.
The predisposition of men to be more prone to impulsively aggressive behaviour is a nice example of their genes making testosterone, for example. While the entire interplay of environment and genes is not worked out, I’d make a personal bet I could take the naughtiest man in the world and reduce him to a wuss if you let me choose the chemicals. The current inadequate state of the art might produce side effects such as rendering him otherwise incompetent to do much of anything, of course. Still, his naughty side could be controlled, both in terms of predisposition to be naughty and the ability to execute any residual tendencies.
And where it’s not a perfect science, it’s because it’s not a perfect science. The problem is not a fundamental inability for chemicals to triumph over learned aggression. We are what we are born much more than what we learn.
Probably for some. At the very least, growing breasts might offer a slight distraction from violent situations. What guy wouldn’t spend at least a few hours with his new juggs?
That said, jail/prison isn’t just about violent people. It’s a tense environment full of boredom. The hormones/drugs might knock the edge off that environment, but in widespread use, they might make it worse, especially if the prisoners knew they were in use. There might be more of a need to show off, prove you’re not affected by them, too. And if you do happen to start getting female features, not only might you need to fight off your cellmates more, but you might also feel the need to prove your manhood more often.
I’d say if given secretly, they’d lessen the degree of violence, perhaps eliminate some of the petty arguments, maybe reduce the fatalities a bit, but not for everyone, and if they were common knowledge, they might actually increase the problems. And it’s unlikely any long-term, widespread drug administration would remain a secret for too long.
Oh great, so now you have a cell block full of bad assed biker types with PMS…this is progress?