For me, the vowel “i” is also different in “five” and “fife.” “Fife” starts with a kind of “uh” sound to me and “five” starts with more of an “ah” sound before eliding into the second half of the diphthong.
That very well may be my accent, as wiktionary transcribes them the same in IPA. It’s the same sort of difference as in “writer” and “rider” in the way my long I is pronounced, so it must be some raising thing.