With visible light there is an aversion response: It is bright so you don’t stare at it. With IR and UV, you don’t see it, so there is no reason to look away. The sun is more than bright enough to cause permanant vision damage, but most people are able to avoid problems because it is painful to stare long enough to do damage. It is very important that devices that limit visible light (sunglasses, welding shades, astromomical filters, etc) also limit the invisible light as well. If only visible light is reduced, the danger may well be increased by using the filter.
A given exposure to IR is slightly less dangerous than the same energy of visible because the eye won’t focus it sharply onto the retna, but the lack of aversion and iris response more than offset this.