How would our political parties kick out a member who wasn't following the party?

Yes.

The party itself cannot kick anyone out, as far as I know, for purposes of the person identifying themselves as a member of that party. But, I think the person could be expelled from the caucus of their party in their house of congress, and could be stripped of committee assignments, etc.

Of course, the big problem with that is potentially driving the person to caucus with the other party, reducing the strength of the existing caucus. If the person in question represents the margin by which one caucus is in charge, that becomes a pretty much insurmountable problem unless and until the margin changes. At that point, the person’s party loyalty becomes less important, and you may prefer having them as a friend rather than an enemy, so there’d be less reason to kick them out at that point, I would think.

There are 4 things being confused here:

  • elected political office
  • membership in a party legislative caucus
  • elected party office
  • membership in a political party

The first one is controlled by the voters, and mostly has to be undone by voters at the next election. Most elected bodies do have some semi-legal processes like impeachment or expulsion that they can use against members, but those are usually difficult to do, often requiring votes from both parties to accomplish. (And can be difficult to maintain – expel a member, force a special election, and the voters in that district can just vote him right back into office. (See Adam Clayton Powell.) Only in impeachment cases does the US Constitution allow prohibiting the person from holding office again, and that has to be an explicitly specified part of the sentence.)

Then party legislative caucus – that is just a group of elected legislators who join together to help each other to get their laws passed. You become a member by saying you are one/voting with that caucus. Can be expelled by the group, but risk is that you will join an opposing party caucus – and vote with them. They can put pressure on a member by peer pressure, removing committee assignments/leadership roles, reducing access to caucus resources (staff, office space), and eventually reducing campaign resources (finances, endorsements) for reelection.

The next one, party office, like a District Chair, is an election by the party members or delegates from that district. Most of these have procedures for removing an officer due to malfeasance, but can be hard to do. (Also, generally the party members or delegates could choose to vote them back into office.)

The last one, membership in a party, is pretty vaguely defined in most states. You’re a member if you say you are. You can give money to a party, or work to elect their candidates, or sign in at party events, etc. – or do none of these, but still say you are a member. But of course, other members can also say you aren’t a ‘real’ member.