I don’t think it matters that you’d see him do it to other women and reduce them to tears. No, that’s not right; it does matter, in that it gives even more validity to your action, but it wouldn’t matter if you’d never seen him assault other women thus. Because he was assaulting you; he was counting on your (presumably) smaller size, and the socialization of women to not engage in violence, to protect him from the consequence of that assault. He surrendered his right not to be punched by behaving as he did.
Yes, lots and lots of times. Never out of anger or necessity, though. 
Not punched, but slapped. Back in college, there was this fellow who was known to have a different personality depending on which substances he’d consumed that day. For whatever reason, he decided to make moves on me. He wasn’t physically threatening, just persistent - kept trying to kiss my neck, then when I shoved him away, he’d announce “She didn’t slap me, she must like it!”.
So finally after the second or third time, I slapped him. Not especially hard, but apparently he took notice. I seem to recall he looked puzzled, then smiled again and went on with socializing.
Now I’m wondering if the guy ever cleaned up. He was mildly entertaining that evening (once I slapped him); I’d heard of him prior to then but I think that was the only time I ever actually met him.
The last time I ever hit anyone was a slap, in college. It was wrong of me - he would not have hit me or any other girl for the world. I admit I was taking advantage of being a woman, and I was just being overly dramatic and stupid anyway.
I console myself with the thought that I was nineteen and stupid and have since apologized to him profusely.
I punched a guy once, but it was provoked. He hit me twice, once when I told him to leave my bank branch, and once again as we left the building. I . . . retaliated.
I also arrested him (citizen’s arrest) and was instrumental in getting him a jail sentence for fraud.
Lucky for me, I didn’t get in any trouble for hitting him; the cops just laughed and congratulated me when the guy complained.
I punched a drunk woman who was slugging my husband in a bar, because he was going to if I didn’t. Man punched a woman, cops and lawyer. Woman punches a woman-chick fight. Not saying it’s right, but that’s the way it is.
I don’t think you qualify as his victim here. 
Many times.