So any form of segregation is okay as long as nobody is harmed by it?
So if I come up with a truly harmless form of separate-but-equal based on race, that would be okay?
Or if you say that’s not possible, why is it possible with gender?
Who would be harmed if we abolished gendered restrooms and locker rooms? If nobody cares, like you say, why not do that? Would anyone complain? Would you declare their complaints invalid because they aren’t really harmed according to you? Why do transgender people care which restroom and locker rooms they use in the first place?
Try as you might, you can’t just make this a simple issue analogous to other civil rights issues as long as it’s laid down over a template of segregation. It’s just not that easy.