Where’s he coming from? He’s learning as he goes along and probably making most of it up, as there most likely wasn’t a poly relationship model for you guys as children. When we’re learning, we stumble. And boy, have I stumbled in my own open marriage. I say open because it’s not a group marriage, but my husband and I are open to see other people romantically and sexually.
Basically, as long as consenting adults are involved, your relationships can look like whatever the hell you want them to look like. No one, mono or poly, can tell you what they “should” be like. Yes, triads, quads or more of wholey invested people are one form of polyamoury. But so is a primary couple plus one or more “extracurriculars.” This is the arrangement my husband and I have agreed to. He is my husband, my primary, my most important person in the universe. I am his. (well, wife, primary, etc.) We both enjoy exploring the energy and connections of other people in our spare time, but such relationships should not and do not take time, energy or large sums of money away from our relationship.
I love seeing him flirt with other people. It makes me smile, makes me happy, makes me adore him more because I’m reminded how desireable he is. He really dislikes seeing me being overtly flirtatious with other people. He doesn’t mind that it happens, but he’d rather not see it. He’d rather be primary when we’re together.
Does this mean he isn’t poly? Of course not. He doesn’t mind when I’m with another man. He asks me who I’m seeing, counsels me when there’s trouble, provides a strong shoulder to cry on when I’m upset with something in another relationship, babysits so I can go out on dates and even lets me know when a good show is happening that I might like to see. He’s 110% supportive of me seeing and sleeping with other men. He’s just uncomfortable witnessing physical affection between me and another. No problem, we’ve talked about it, and I’ve decided that this isn’t a deal breaker for me. I just don’t flirt with folks when he’s around.
For obvious reasons, we’ve never had a threesome. 
And that, of course, is where the similarities between my situation and yours end. Your bf has been sexual with both of you at once, It would be logical to assume that the two of you were physically intimate in his presence then, so logically, it shouldn’t bother his to come into the living room seeing you snugly on the couch.
But when has emotion had shit all to do with logic, EVER?
And of course, this is further complicated by the fact that he’s living with his extracurricular, not his primary. Which is unusual and pretty hard to deal with emotionally, I imagine. I suspect that when she’s not in town, you feel much more like his primary, no? And pine for her at the same time? So being asked to give up your status as lady of the house, being his pseudo-primary and your lovely girlfriend all at once seems a little, well…Cinderella *before * the ball.
There’s two questions here. A. Is the way he is and the way he feels the way he *wants * to feel? That is, is he exploring what’s going on and working through something, or is this it, end of story? B. How do you feel about that?
1A. If he’s going through something, and you feel like helping him work through his process, then by all means do so.
1B. If he’s working through something and you don’t feel like helping, don’t.
2A. If this is the way he wants to feel, he feels totally justified in feeling, and wants to feel forever, and you decide it’s OK with you then by all means stay.
2B. If this is his feelings forever and you’re NOT OK with it, leave!
ahem I suspect 1A or 2B is the case, seeing as you wouldn’t have posted here if you were truly OK with it but didn’t want to help.
You can be not-the-primary without being a victim. I think you’re doing that very well so far, and I wish you strength and love as you search your heart for what’s best for you. 
