I can’t think of anything that literally fits your requirements, but there are several works in which very alien-thinking aliens appear, with at least some suggestion of why. The Star Trek: The Next Generation episode with the alien captain whose language is entirely based on metaphor, for instance. Picard knows what he’s saying – the words are translated – but the thought prcess and meaning are harder to fathom.
Another work with similarly odd alien language 9which was worked out in some detail with help from a linguist, but which we don’t actually see much of) is in Harry Harrson’s West of Eden and its sequels. The intelligent dinosaurs have language that operates on a completely different set of principles that human language, and one human ends up being killed because she can’t grasp even the basis of the language. Herf brother, who’s younf enough to not be set in his linguistic ways, grows up able to speak both dinosaur and human, which is apparently not a trivial achievement. There’s an appendix in the book explaining the language.
Some works discuss alien intelligences so different from ours that we cannot really understand them, although they’re not clear on how those minds work. That’s the entire basis of Stanislas Lem’s Solaris and of Terry Carr’s short story “The Dance of the Changer and the Three” (circa 1969).
If you’re looking for other alien characteristics, look up Hal Clement’s work. His aliens almost always have very human thought patterns, which is direct opposition to your query (and which always bothered me a bit. Clement’s aliens are always significantly different from humans, living at much higher or lower temperatures, or in different atmospheres, or higher pressures, or lower light levels, or something significant. He hadles these differences with considerable insight and scientific accuracy, but you’d think they’d impose huge psychological differences as well. Perhaps in the interest of focussing on stories and interactions, Clement minimizes such differences). But you can see some startklingly different features of aliens that are REAL differences that you could easily see leading to different way of thinking, even id Clement doesn’t actually depict different psychologies. And this seems to satisfy the second part of your request.
In one story, Clement tells about a nearly airles world with creatures that “see” by having pinhole-camera-type smell detectors. They have globelike structures with pinholes, and molecular sensors spread, retina-like, on the inside opposite the hole. The mean free path of molecules is high, so any outgassed molecules travel in straight lines and paint a smell “picture” on that retina, so the creatures literally “smell” images. It’s the most completely and accurately described alien sense I’ve ever encountered, and it would work. Unfortunately, none of the creatures who have this are intelligent, society-building beings.
Another story tells of a human crew who find an alien spacecraft. It’s one long abandoned, but they can glean from it that their technology and their entire approach to technology is vastly different from ours. they don’t use tools for forcing, as we do. No wrenches tightening nuts onto bolts, for instance. They use interlocking ovals, and catalysts for bonding and releasing, and electrorestrictive forces for making radius of curvature changes in metal faces. We never meet any of these aliens – only their works. But he has convincingly described a different approach to technology that would undoubtedly reflect a different psychology.
Come to think of it – Harrson’s intelligent dinosaurs have a very different approach to technology as well. They have no fire and no written language. They have practically no material culture using metals, wood, or stone. Instead, they create useful items like “cloaks”, “microscopes” and the like by performing genetic engineering on other animals. their cloaks are living sheet-like creatiures. They microscopes are distorted frogs, whose own eye lenses have been reconfigured as objectives, eyepieces, and condensers. Exactly how they perform this genetic manipulation without a material culture in’t explained.