Surprisingly enough, Google has been quite unhelpful on this; none of the top links seems to really answer my question.
I ask because I only relatively recently got over a cold that gave me a fever for three days, so I’ve been a tad paranoid. So when I bumped the back of my hand against a bus hand… uh, handle, then used the back of that same hand to wipe my eyes a few hours later, my mind immediately jumped to how miserable I was the last time around. 
Thanks!
Mayo Clinic says maybe - I think. Basically that cold and flu viruses have differing lifespans that can last from a few minutes to 48 hours, and generally hard surfaces like metal or plastic lead to longer viability than soft surfaces like fabric.
So the answer to “how long do germs hang around skin” is… maybe… depending on whether it’s metal, plastic, or fabric?
Thank you, Mayo Clinic!
How long do they hang around? I’d imagine they stay forever. Why would they leave?
Maybe the question you want to ask is how long they stay alive where they are, or how long they remain infectious. And the answer to that will vary depending on the specific germ in question/
It is a bit like asking “How long is a piece of string?” There is no universal answer. Many viruses and bacterial spores can hang around for a very long time. They don’t go away on their own. That is why we wash things. On the other hand, you are not likely to get an infection just by picking up one or two.
And for many viruses and bacterial spores, that will also be forever, to most intents. But it is not a reason to panic. Humankind has been living with these things effectively forever too.
Well, in that maybe a given germ might last a few hours, as you reported worrying about in the OP 
and that germs have a longer lifespan on hard surfaces than soft. So the bus was surely full of germs, but did they last long enough on your skin to get transferred after a few hour delay? Maybe.
As an interesting exception to that, copper apparently destroys influenza viruses among other microorganisms. Wood is also good at destroying bacteria; I’m not sure about influenza though.
If you really want to slow the transfer from bus to you, teach yourself to not touch your eyes, nose or mouth until you have washed your hands.
I usually do, but I forgot, since it was just the back of my hand, and I usually don’t touch anything on the bus anyway.