Well, major differential background cues (e.g. mountains versus ocean) make it really obvious, but only if you’re familiar with them. Bring in someone who’s not from around there (e.g. the west coast), and that association between ocean and sunrise doesn’t exist for them.
Alternatively, there are places where such blatant cues don’t exist - for example, many places in the Great Plains region of the United States, which is a large region that doesn’t have a lot in the way of acute elevation changes. Stand in the middle of a square (mile) of farmland in Kansas, Nebraska, or South Dakota, and you won’t see much difference no matter which way you face. Sure, if you study the scenery, eventually you’ll spot minor differences in the treeline or farm buildings, but if you move one square over those cues will be totally different - in other words, there are no visual features around you that are intrinsically and uniquely related to sunrise or sunset, or east or west. In spite of that, I’m pretty sure that sunrise and sunset would be perceived differently by an observer there.