Not particularly dangerous? Careers have been ended this way. It was worse in the days before batting helmets (Ray Chapman was killed by a pitch in 1920), but even now a guy can be really badly hurt. I saw (on TV) Terry Steinbach get hit in the head so hard it broke his helmet - he was out with a concussion for a couple of weeks (if it happened today he would probably be out a lot longer).
Here are some more cases where batters were really hurt by pitched balls:
2005, Adam Greenberg hit in head, career ended due to vertigo
2000, Mike Piazza hit in head, concussion
1984, Dickie Thon hit in face, broken orbital bone, partial blindness
1967, Tony Conigliaro, hit in face, fractured cheekbone, never the same again
1979, Gary Roenicke, hit in face, 25 stitches
1995, Kirby Puckett, hit in face, broken jaw and vision problems that eventually ended his career
There have been plenty of cases like this. There has been an even greater number of less-serious injuries, such as broken fingers, bruised ribs, damaged wrists, broken ankles. And even when nothing is broken, it hurts.
Yes, the players have good reflexes, and they usually can get out of the way of a pitch. Sometimes they can’t. The pitch is coming in very fast, and they have little time to react. Sometimes the pitch curves toward the batter so that jumping backwards doesn’t help.