Does your son care? If he doesn’t care, you can’t teach him. You seem acutely aware of the awkwardness, as do his siblings, but you mention he doesn’t know how he looks to other people. If nothing about the situation bothers him, there is absolutely no way you or his sibs (or his teachers or the other kids at school or…) can make anything stick.
If he’s frustrated by how other people treat him, but doesn’t seem to be able to pick up on the smaller negative reactions that come before the really big ones that upset him, then you might be able to do something.
It’s possible he just doesn’t pay enough attention to other people to pick up on their reactions. This problem is likely to be self-fixing as soon as he discovers girls (or boys, or both, as the case may be). If he’s also known as a particularly bright or talented kid, there may also be an element of resentment, as he recognizes that when he and the rest of the world come into conflict, the rest of the world blithely insists that he learn their social language, while they make no effort to learn his. He probably thinks that’s deeply unfair, and he is correct, but it doesn’t excuse him from having to go along with the plan anyway. When he wants to get someone’s positive attention badly enough, he’ll learn.
If he’s always been awkward, and the other kids have consequently always treated him the same way, then he might be stuck in a self-perpetuating cycle: His cohort sees him as the “weird kid” and interprets anything he does as “weird” no matter what it is, and since everything gets the same reaction from the other kids he has no opportunity to learn socialization the normal way, i.e., by making boneheaded mistakes and learning how to fix them. He may just need someone to spell it out for him, since the lack of feedback makes it very difficult to figure out the rules inductively.
If that’s the case, it might be a good idea to find him a particularly nerdy counselor who specializes in working with autistic kids, whether your son is actually on the spectrum or not. Most people learn how to get along with others pretty much by trial and error. Do this and people are happy, don’t do that and people get ticked at you. Consequently, they’ve never had to develop conscious awareness of what social signals look like independent of their meaning. You ask, “How did you know he was bored with what you were talking about?” and they say, “Because he looked bored,” without being able to tell you what exactly bored looks like. The people who work with kids on the autism spectrum have experience taking these things that most humans interpret instinctively and translating them into descriptions that can be interpreted analytically instead.
I wish I could get more specific with resources – I do a lot of writing about people-reading and trying to put social skills into words – but I don’t know your son, and without more details I’d just be spitballing kind of at random.
)