Before the introduction of the word ‘orange’ to English, The colour was described as a shade of red or scarlet - there’s probably no better example of this than one of Britain’s very, very few native orange flowers, the scarlet pimpernel.
The Irish language has three different words which are translated into English as “green”. But they are not interchangeable, and to us they do not describe different shades of the same colour, but different colours. English-speakers describe all these colours as “green”, and presumably regard them all as shades of one colour.
So I think it’s not just that people don’t recall colours for which they have no name; they don’t even see them. (As in, register them, observe them.)
I think we’re indeed extremely influenced by cultural expectations. We “know” that the default colour of the sea is blue. But in fact, if you observe it objectively, it’s rarely blue. And we marvel at the beauty when we see an actually blue lagoon. Deciding that the sea is black doesn’t seem particularly outlandish to me.
Another example of it is the sun always being depicted as yellow, even though objectively it isn’t.
On all other nights we eat leavened products and matzah, and on Wednesday night only matzah.
On all other nights we eat all vegetables, and on Wednesday night only bitter herbs.
On all other nights, we don’t dip our food even once, and on Wednesday night we dip twice.
On all other nights we eat sitting or reclining, and on Wednesday night we only recline.
I don’t know if that is the same. The sun, when it is high in the sky, is brilliantly, obviously white. But you can’t look at it for more than a split second at those times. But when the sun is low in the sky, it is indeed yellow. Then orange, then red, and then gone. So I think we often depict the sun as yellow because the only times we can look at it without extreme pain, it is yellow, or darker.
It’s also up against a light blue background, so the contrast can make it seem yellow.