I hate the fact that when I die I and there is no afterlife I won’t be able to tell all of the those people who do believe in an afterlife that I was right and they were wrong. 
More seriously, I was raised a wishy washy Methodist, who finally lost the last bits of belief in a higher power when he read about the anthropic principal, which eliminated the lingering doubts of the improbability of intelligent life.
But I still occasionally pray about things that worry me. I recognize intellectually that it won’t change anything but it gives me something to do when I can’t do anything else. So I would say that the worst thing about being an atheist is that there is final entity to go to to take care of the things you can’t.