The title pretty much says it all. What’s the origin of the expression “He’s a nutter/nutjob/nuthead”, in the sense of calling someone crazy?
You need to start with the meaning of “nut” to indicate the head.
From that meaning, it’s only a short hop to making a disparaging remark about someone who is slightly “off” “crazy” etc.
Might the original nut related term for someone crazy possibly be “nutcase”? A nut case (as opposed to a nut in its case) is something hollow, making the connection to mental deficiency a short step, especially when combined with “nut” as a term for “head” as shown by samclem. The other terms could have been derived from “nutcase”.
“Nut case”, AFAIK, is just a variation of “head case”. Literally meaning medical cases that are related to the head (cf “mental case”). I see nor reason to ascribe any different origin to “nut case”. It’s just another case where “nut” has been substituted for “head”.
I’d have to see some evidence before I believed that “nut case” has ever referred to a nutshell. I have never, ever seen or heard it used that way.