I’ve often heard the same about the fishermen in the west of Ireland.
The answer to the OP may be that for many island-dwellers, the sea is a dangerous workplace, and they would not dream of going there for recreation, any more than the coal miner goes down the mine for pleasure when he’s finished his shift.
More on the Balinese… religious temples are oriented in specific directions. The mountains are sacred, the sea is where evil spirits dwell. In each village, the temple for cremations is the one located closest to the sea and after the ceremony, there is a procession to throw the ashes into the ocean.
Back in the 60s, Kuta was just a beach and palm trees, no development, no village. One could be grateful that the ensuing rampart development of the coastal areas did not displace the local inhabitants.