Transportation - if the stone was not near the place they would use it, or a water way, then hauling it was expensive.
I read somewhere that around 1700 wood was cheaper to bring from the Baltic to Britain than to haul it sixteen miles inland. Canals and then railways made things more flexible.
Of course Stonehenge could be viewed as an exception 