Are you Scottish? I know that Scottish doesn’t have the fir/fur merger. So it wouldn’t be surprising if you keep distinct vowels in other words, too.
That said, I do maintain a difference between schwas. I at least have two, one for I and E and one for the rest. I also sometimes have a different one for O and U, where O is a rounded schwa and U becomes short oo as in book or the French schwa (which is fronted and rounded).
Because, really, what happens in English is vowel reduction, and it doesn’t always go all the way back to the original schwa.