I think that without an express “no” or something similar it can’t be rape, at least in situations where people are capable of talking and understanding. I’m a big guy (6’6" 350#) and I’ve never had been with a girl who was afraid of telling my no or implying it. Or I should say that I’ve never known about it.
I think unless there is a deep seated fear there is plenty of time during transition phases to say “no” or what I hear “Not yet”. It takes time to take off two people’s underwear and it has been used many times in my experience to stop the action. On the other hand I think that it infantilizes people if both parties have to stop at every transition and ask “Are you sure?”
I’ve been a similar situation to the rugby player case. In college we were trying to get some girls to play strip poker (3 girls and about 10 guys). When we finally convinced the girls to play they said they wouldn’t play when they were so outnumbered by guys and made all but 4 of us leave the room. Sure it wasn’t 4:1 but I think if those girls spoke up it is reasonable to expect other girls to do the same, I mean really how hard would it be to tell the first “two I’m going home unless you throw your mates out”. Every guy I know would get into as many fights as necessary to clear the room to continue the action.
Like you brought up, if a person is so genially afraid for their safety that they won’t speak up and give some idea that they are not a willing participant, how can we trust that we are getting a voluntary “yes”? The way I look at it at some point you have to trust the person you’re having sex with either to stop you and say they don’t want to do something or that they won’t lie to you when you ask. Personally as long as they seem into it I assume the first and if they start seeming hesitant I assume the second.