UPDATE:
I recently was rehired by the company last week, and on the very first day back he offered me a drive home (this was with other people I work with, a carpool) anyway, I point blank refused to accept a drive home. Same thing happened the next day, luckily I ride to work on my bike now.
What happened next was one of my friends explained to the guy why I never wanted a lift from him, and told me that they had a few laughs about it, I don’t mind about that, the guy doesn’t speak to me anymore, my friend said that he doesn’t wanna bother trying with me.