It’s only limiting if you want to go there and back in a human lifetime. Once/if humanity learns to live in space, then even at the slow pace of migration, you still get a galaxy filled with life in a matter of millions of years, which isn’t long in the grand scheme of things.