I think one way an alien species could think differently than humans would be to have aliens that instinctively understand odds and chance correctly, the way the humans only do if they’ve studied mathematical probability and force themselves to think logically. They would correctly regard the human belief in “luck” as delusional, and would behave differently than humans as regards hazards, based on a more rational cost/risk assessment than most people use.
On the non-SF front, you might want to check out books on the anthropology and social psychology of time, such as The Dance of Life by Edward T. Hall or some of the books by Eviatar Zerubavel.
I’d take a look at some of Stanislaw Lem’s sci-fi books, especially Fiasco and Solaris, both of which treat the subject of humans encountering vastly alien and incomprehensible organisms. Both books do a great job of portraying basically futile attempts between humans and aliens to find the most basic common understandings to use to communicate with each other, and the harm that they do each other (usually unintentionally and unknowingly) while trying to figure it out. Lem’s aliens are truly alien – the reader never gets a chance to “figure them out”, but they’re presented in a way that makes it believable that they’re acting with a consistent (but inscrutable) logic.
The Ascian language from Gene Wolfe’s Citadel of the Autarch. Every sentence they use is drawn from a memorized lexicon of “approved” statements. Meaning is implied by context.
The Charoneans in The Ring of Charon and The Shattered Sphere by Roger MacBride Allen; the Arachnids in In Death Ground and The Shiva Option by David Weber and Steve White; the Phinons in Dykstra’s War. Intelligent, in many cases, yet completely instinct/program driven. They attack humans not out of malice or ideology, but simply because they are made that way - it is literally inconceivable to them not to. The Arachnids, for example, end up ( justifiably ) on the losing side of genocide, yet never once wonder if their unprovoked attack on humanity or the Star Union or the Orions was a good idea - or even a bad idea.
In George Orwell’s 1984 he talks about how the language of Newspeak is designed to phase out concepts from the people’s understanding.
I can’t remember the name of the story, but Harlan Ellison has one where a few humans are imprisoned (each in solitary confinement) by an alien race that the humans are at war with. Their captors torment them by dangling the possibility of escape. They arrange situations that look accidental and where there seems to be a chance of escape if the human does everything just right. Every time it turns out to be a deliberate setup and the humans are punished for trying to escape, but they have to keep trying–hoping that this time the aliens really did mess up and leave a way out.
Eventually the main human protagonist gives in completely. It then is revealed to him that he is really an alien who volunteered to be physically altered and have his memory replaced with that of a real captured human so that they can understand the concept of surrender (something that humans do that they don’t do and can’t quite grasp) and find a way to teach it to their race. It seems that they are losing the war and must learn how to give up on a course of action (something that is simply not in their nature) or they will keep on fighting until they are extinct.
The elephant aliens in Larry Niven’s Footfall. They evolved from herd animals. Their goal is to get us to surrender because by their mindset, once you’ve surrendered, your completely subservient. It’s instinctual with them.
So when we pretended to surrender in order to gain advantage for our attack, they were completely baffled. The really cool part was, even though they knew we broke our surrender, they still couldn’t overcome their instincts to break their own surrender.
And the ones who are most fanatical on this point also give the Romans a free pass to set up equipment unmolested one day out of every week…
I think Orson Scott Card’s Ender’s Game and Speaker for the Dead story arc qualifies. It’s kind of the classic clash of civilizations where there is no comprehension of the enemy’s motives by either party.
Indeed, Ender becomes a protector of “Alien Species” after his near decimation of the enemy in Ender’s game.
I think the Puppeteers are even more alien; they regard themselves as abject cowards. I recall a scene (from Ringworld, IIRC) where Nessus (who is considered insane by his people because he is able deal with humans) is in his equivalent of a holodeck, dancing with a crowd of other simulated Puppeteers to maintain what sanity he has left.
The Left Hand of Darkness by Ursula K. LeGuin has humanoids that are usually androgynous, but periodically come into “kemmer” for sex, becoming more or less randomly either male or female. (Their big perversion is having yourself surgically fixed into kemmer in one gender or the other, which causes major problems for the human male assigned to contact them.)
In The Sparrow (and the followup Children of God) by Mary Doria Russell, a fundamentally different mindset on the part of two related alien species leads to disaster for the human contactees, as well as societal disturbances for the aliens.
Going into more fantasy than sci-fi, there’s a series by Raymend E. Feist and Janny Wurts: Daughter of the Empire, Servant of the Empire and Mistress of the Empire. In each, the main character deals with the cho-ja, a race of large, highly intelligent insect-like creatures. She eventually meets with their queen and arranges an alliance with them. The relationship between Mara, various cho-ja workers and the queen are very well-done, fraught with risk and misunderstanding.
David Brin wrote a short story, “Just a Hint”, showing Earth and an alien world similar to ours in parallel. The point of the story was that each planet was facing a crisis that the other planet had resolved. So there was an obvious solution to both problems but each society had a blind spot that kept them from seeing it.
Brin wrote another story, “Sshhh…”, about Earth being discovered by the intergalactic federation. Most “primitive” races like Humans underwent a prolonged period of feeling cultural inferiority when exposed to the advanced alien civilization. But a rumor was started on Earth that we had a quality that no other race in the universe had and that, while we going to have to learn a lot to catch up, we would also have to conceal our unique ability so we wouldn’t embarass the aliens for their lack of it. The story never said what the quality actually was and implied that its existence might have been invented to prevent humans from feeling completely inferior.
Harry Turtledove wrote a story, “The Road Not Taken”, based on the idea that faster than light travel was actually a simple technology. Most races invent it around the time they start making iron. But humans somehow missed the obvious and because of that we had to go on and discover all of the other science and technology we have.
Good recommendation. The sequel isn’t as good as the first book, which is amazing, but it’s still worth a read.
I read a story about aliens that had no sense of proprioception, i.e., they had no sense of limb or body movement or position. They had to constantly use visual feedback to stand, walk, or sit down. Visiting Earth, they were amazed by dancing and gymnastics. They would fill up sports stadiums to marvel at the ball players. Controlling their own bodies felt like controlling puppets and, indeed, their word for suicide or being killed translated to “cutting the strings”.