Yep. People have done that sort of vocal abbreviation of numbers for hundreds of years. There’s an Italian term (in art-history?) I’m having trouble remembering, which seems literally wrong on its face, but is just a contraction of another number.
Anyway, in tennis, it seems like the following is probably true (though I admit I’m making this up–extrapolating from what little I know, and ignoring possible rules changes–over 600 years–about which I don’t know):
Hundreds of years ago, in French, a game of tennis was scored to sixty. In order to win, you had to at least complete “soixante” (60) (and of course, lead by two points as well). “Soixante” was actually four points. The number 60 signifies a complete game–as it evokes the complete hour–better than the number four. It also echoes the six-game set. So the first three points of a game were called “quinze” (15), “trente” (30), & “cinq et quarante” (45).
At some time (I don’t know when), any number higher than 45 came to be seen as unnecessary. Once a player reached sixty, one of three situations applied: the game was won, it was “deuce,” or it was “advantage.” So “sixty” was not kept as a term. Any number higher than sixty was extraneous–extra points were just there to get that two-point lead, and win the game.
Still hundreds of years ago, and probably in French, not English, “cinq et quarante” became just “quarante.” There was no possibility of confusion; there was no other “fourty” in the game. And so it went on that way, and was translated as such into English.
By the middle of the Twentieth Century, English speakers were shortening “fifteen” to “five”–again, a shortening of the word. But I suppose that French speakers would find it absurd to say “cinq” (5) for “quinze” (15), as there’s no shortening of the word in speech, but rather a transpositon of two consonant sounds.