is evolution a fact?

Math is the language we use to describe the universe. The universe doesn’t “care” about math. It’s a human construct.

I studied physics, and I agree. But does that say anything fundamental about the universe itself, or our ability to understand it?

Only in the sense that sentient beings should understand this. Again, the universe just is. It doesn’t “need” math.