Yeah. By that sort of argument, we should be completely unconcerned about the potential dangers of major asteroid impact or nuclear war. At the very most improbable worst, all they could do would be to wipe out most multicellular life forms, and multicellular life forms have only existed for about one-quarter to one-fifth of the earth’s history. So on average the change would be trivial.
To the OP: Saying that high CO2 levels aren’t a problem now because they used to be much higher back in geological time, before mammals even existed, is sort of like saying that when your house is on fire you shouldn’t bother calling the fire department. Because back in prehistoric times there were fires sweeping through your locality every few years or decades, so on average, getting burned to ashes is completely normal.