There are a lot of great answers above, but the simplest answer is IMHO the biggest:
Change causes upset
When you introduce something new into an ecology and it happens to have an advantage, it makes a big change to the local ecology, and the results are unpredictable. A new predator without a native enemy can flourish as it eats the prey, until there are no prey left, after which the predator population declines rapidly (either entirely, or it shifts to a new niche, a new prey). Meanwhile, the prey population is gone, which affects other populations, sometimes dramatically. It’s easy to get a domino effect, and the result is an ecology that changes rapidly for quite some time until it reaches a new equilibrium.
Meanwhile, anything that lives in the ecology is at risk, including us. I don’t mean our survival is at risk, but aspects of our economy and comfort and leisure activities may be.
From one point of view, man’s introducing a new species into an ecology is no different than a natural accident doing so (such as a raft of land animals arriving on an island.) The new species is either successful or it’s not, and eventually, things get worked out. However, in the meantime, a lot of species can get wiped out. If humans live there, they might depend on some of those species.
But a big difference between natural accidents and human-caused introductions is the rate and scope. Likewise, the rate and scope of ecological upheaval will be great.
In the long run, things will definitely play out. But in the meantime, it makes perfect sense for us to do whatever is reasonable to avoid rapid and possibly deleterious (and definitely unpredictable) changes to local ecologies.
This is true regardless of how much of a tree-hugger you are, or how much you thing biodiversity is valuable (with which I certainly agree). It’s the “conservative” thing to do, using that term based on its generic meaning, that is, someone who doesn’t want to try something new and likely to be risky, without a very comprehensive argument in favor. (Please forgive my using a noun definition for an adjective, for brevity.)