Having done a lot of breaking - boards, concrete, rocks, bricks and so forth on my way to black belt (got it, too), Here it is: it’s physics, but not just physics.
The way I explain this to my students is, the force you hit with is proportional to the mass (your hand, foot, whatever) and the speed at impact. You cannot do much about the mass of your hand, but you can learn to hit faster, and thus harder. kinetic energy = (mass * velocity * velocity ) / 2 so there is more to be gained by getting faster than there is by making your hand bigger. (you get faster by getting stronger, so more pushups!)
So much for physics. Another very important aspect of this is that your hand (or foot, or head) has to be conditioned over time to do this. This is covered by Wolff’s Law, (another google exercise, or just look here: Wolff's law - Wikipedia ). Briefly, bones that are made to work hard get more adept at working hard. This is why your hand hurts from breaking one board and stepping up to five or more right away. Bones will adapt to the load put upon them, but it takes time to for them to do it.
Here is the thing I spent an entire sunday afternoon on once: When learning to break concrete pavers (2 inch thick by 8 inch wide by 16 inch long), my instructors were trying to get me to hit the paver off-center, rather than at center of span. Why? This really is just physics, but what is the explanation for hitting off center if this is correct?