For me, it was when I realized that I was angry the majority of the time and that I just didn’t want to be angry all the time anymore. I went to therapy. It helped tremendously. (That and the Wellbutrin.)
I think the best way to decide whether you should do something about it is to ask yourself if anger is interfering with your life or even just your wellbeing. Then ask yourself if you want to do something about it. If either answer is yes, do something about it.