When does hanging out with your ex girlfriend considered foul?

I guess I was just being emotional and insecure about my place in our relationship, knowing that he values their friendship more than our relationship.