Look at it this way. A majority of the people in a Congressional district approve of a bill, but their representative doesn’t. Forget about fairness. The real question is one that has been debated since the beginning: should a representative vote according to the majority or according to a personal understanding of what it best?
There is no obvious answer to this, and no obvious way to apply it to all bills. It may be best to vote one way at times and the other way at other times.
There is also no obvious way to judge the representative. You can certainly disagree with the voting on any one issue. The question then becomes how to weigh that disagreement in the context of all the issues that arise over an entire term. For most people, that one issue has to be extraordinarily important to override general agreement on other issues.
Now apply this to the case of the Speaker. He or she is the elected representative of the majority party. He or she has been tasked to determine what is best for the party as a whole. If the majority of that party has sufficient cause, they can vote in a new Speaker, but that cause better be sufficient as all hell because anything less will undercut the authority of the new Speaker. Until that time, better to go along with the Speaker because that is the entire point of representative government. If you want the majority to rule on every single issue with no overall guiding principles, you need to move to direct democracy. But nobody sane thinks that can actually work. And certainly nobody in representative politics thinks it can because they are part of an extremely successful system that has thrived for centuries allowing representatives to represent rather than merely serve.