The very word is the reason, I think. It sounds like a nonsense word in the vein of ‘doolally’ and ‘wacky’ etc, which aren’t real words at all, but still refer to the effect of being ‘nuts’.
Let’s quit monkeying around here, many words sound crazier than bananas. Zenster sounds closest in suggesting “going ape for bananas” but I swear that the phrase I have heard is “going bananas” Doesn’t sound right Zen.
I was told by a person who had served in India during WWII that going doollally was a term that was coined by troops stationed over there.
The full term was “To be tapped with the Dolali hammer” meaning to go slightly mad.
He claimed it was a referance to a large military camp where soldiers ended up with a mental institution there.
When I visited India myself I went to Bombay and was assured that there was a place of that name in the area(since India is huge I took that with a pinch of salt) and that it had been a British army depot.
This could all be total tosh so if you know more I’d appreciate the telling.