The only thing I can think of that is regularly compared to the size of a peanut is a person’s brain. In that case, I imagine a cartoon peanut (shell and all) inside the person’s head.
On the other hand, if you said something like “a chickpea is the size of a peanut”, I’d think you meant without the shell.
If someone says “peanut”, without giving it any more thought I envision the “tasty inner nugget”.
Well in your case I’d expect as much…but you’re obviously biased 
The discussion last night revolved around one of our cats doing something stupid, after which I referred (lovingly, I assure you) to his little peanut brain.
Peanut I think the size of the shell because that is the way I usually see them.
Pea I think of the small green ball because that is the way I usually see them.
A peanut is bigger than a pea.
Another vote for the whole shell. Like hogarth says, usually a cartoon peanut in fact.
It’s not a clear win for either side (as the poll results indicate.) It’s often context sensitive. A wart the size of a peanut is bigger than a pea, but not an inch long. There aren’t many things I’d compare to the size of a peanut shell, due to it’s odd shape. Maybe a caterpillar or a beetle.
My gut reaction is whole shell, even if individual nut does seem more logical.
If the size is being used for general reference as in “his package was the size of a peanut”, I would just assume it to be referencing very small.
If I were describing very specifically a thing of a certain size, I would maybe use bean instead of peanut if I were talking about a single nut. If I were referencing something the size of a peanut in it’s shell, I would likely reference the size of a thumb, or child’s thumb.
Semantically speaking, I guess with the shell on it’s (usually) a pair of peanuts and some casing.
Perhaps the next step up is circus peanut?
I think of the size of a shelled peanut–like out of a can of nuts.
I’ve noticed a lot of people nickname their unborn children “Peanut” in utero. I always assume this is because when they went in for their first sonogram, when they asked how big it was the doctor said “the size of a peanut.” They think “Ooooo, it’s a cute little peanut” (when in reality it’s a terrifying parasitic fish-monkey) and then a few months later they’re talking about how “Peanut” is really kicking today. They don’t realize that there are 3 other people on my Facebook wall with the same fetal nickname. When they say that, I imagine they mean “the whole peanut,” because a whole peanut is shaped sort of like a fetus.
the peanut-shaped styrofoam packing bits are shaped like a whole peanut, shells and all.
It’s a tough call, but I’m leaning toward the individual nuts that you’d get in a jar . . . only because that’s the way most people get them (unless, like me, they’re addicted to Five Guys).
I picture the individual nuts, but I think picturing the whole shell is more logical, for a reason given above: there’s otherwise no real difference between peanut-sized and pea-sized.
The entire ground nut, shell and all, beause otherwise it would be the size of a peanut kernel.