In other words, you mind.
My advice, as someone who is currently in a poly relationship and has been for 5 years and counting, (I don’t really feel like going into a lot of specifics on here, but my situation involves double relationships but nothing casual, so we may be atypical; I’m not really into the poly “scene”) is to fix this situation immediately. Things that bother you a little now will not magically fix themselves. They will not. You are telling yourself, “Well, I am bothered by this, but not TOO much, and I’m sure it will get better eventually.” That is not true. It will either remain the same, or get worse. Your husband has zero incentive to work towards changing his feelings on this issue. What I can see happening is resentment building up and eventually poisoning either the poly aspect of your relationship, or the entire relationship.
What you really don’t want to have happen is for your husband to start “messing with” a girl that he then establishes an ongoing relationship with, so that then if/when you decide that his double standard is untenable and you would prefer he not mess with other women, this will mean breaking apart a relationship that at that point may hold great meaning to one or both of them and may be quite painful to end. Not that you should necessarily let that stop you if that’s what you need in your marriage, but that’s a lot of guilt on your shoulders that you can avoid if you just solve the problem right now.
If I could only give one piece of advice to people considering a poly relationship, it would be to never let resentment fester for even a day. With multiple relationships, there are so many possibilities for hurt feelings and misunderstandings and trampled boundaries. When that starts to happen, you need to speak up and put an end to it right away, whatever that means for you.