For the mostpart, I get where you’re coming from, except for what I quoted, but this is not a situation where you can have a huddle and discuss a mutually agreaable course of action. A decision has to be made on the spot, and anyone who doesn’t agree with that decision can alter their own actions accordingly - in your case, he makes things physical, and you keep walking.
I was in this very situation many years ago. My friend and I were walking our dates home from a party, and passed by a bar with a bunch of drunk guys standing out front. One of the drunks had some inappropriate words for my date, which we ignored, and continued on our way. Handled perfectly.
…except that he followed us, and continued running his mouth. I turned to the other three in my party and said “Keep walking, I’ll catch up.”, then stopped and turned around to let the drunk catch up. My plan was to stall him briefly, put some distance between him and my friends, and then jog to catch up to them. He bumped into me agressively, at which point I said “That’s enough man. Go back to your friends over there, and we’ll be on our way.” He shoved me, and expressed an interest in fighting, which I politely declined. As I turned around to run and catch up with my friends, I found they were standing right there. :smack:
On one hand, my friends’ not following my instructions made matters worse, on the other hand, who am I to tell my date and her friend what to do in such a situation. I any case, their failure to follow my lead eliminated my escape option. The drunk had more friends nearby than I did, and I don’t think a fistfight is warranted in this situation, but the drunk kept shoving me and telling me to hit him. I walked backwards with each shove, until I ended up backed against something, at which point I shoved him back as hard as I could. He reeled and staggered from the shove, into another passing drunk. Passing drunk WTFs mouthy drunk, and a split second later, they’re punching out their differences, and we are happilly on our way. The police were on scene before we reached the end of the block.
My date thanked me for handling things, and I may have gotten a smooch out of it. 
To answer the OP: Words are words - keep walking, but if the agressor escalates things, the level of response increases proportionately.