This is mostly gleaned from Iona and Peter Opie, esp. A Dictionary of Superstitions.
The belief and custom are common to Britain and its colonies; the earliest attestation is only 1919 (“To secure good luck of some kind, usually a present, one should say ‘Rabbits’ three times before going to sleep on the last day of the month, and then ‘Hares’ three times on waking the next morning.”) The Opies connect it to the custom of bowing to (or otherwise paying reverent attention to) the new moon, which goes back to medieval times (for bowing) or ancient times (moon ‘worship’).
As to why rabbits, whole books have been written on the subject. My personal favorite is George Ewart and David Thomson’s The Leaping Hare. In essence, rabbits are fertile and therefore lucky creatures, and what we see as “the man in the moon” is perceived as a hare in many parts of the world.
There are other ways of marking the new moon in contemporary folklore. I was woken up with “a pinch and a punch on the first of the month,” though to be fair they were fairly gentle.