Phew, well, I knew we’d get to this sooner or later so I might as well have a pop at it now.
This is perhaps the most difficult question one can ask of a physicalist like me. I believe that everything is physical or, at least that everything supervenes on the physical. And the trickiest thing to translate into a physical entity, an arrangement of spacetime, a “bunch of atoms”, is mathematics. So here goes.
The universe (at least this region of the universe having three dimensions of space and one of time) is so. It is how it is. It does not suddenly disappear occasionally, nor does gravity go repulsive at the drop of a hat, nor does the entropy of things begin to randomly decrease. This is how the universe is. We have words for “how it is”: Rule or Law or, in a scientific sense, Theory. These things exist insomuch as they are the characteristics of the universe.
Now, we humans (and even dogs to a certain extent) are able to do something called “learning”. Characteristics such as “red”, ie. electromagnetic radiation of wavelength around 630 nanometres, form chains of neural dendrites in the kilogram or so of offal residing in our skulls. These strings of atoms which encode input are called memories.  After forming a “bedrock” of simple, immediate characteristics like “red”, more complex ones can be built up. For example, a toddler or a puppy who sees a ball being thrown has not yet formed a firm enough set of neural strings to be able to encode this characteristic of the universe usefully. After years of observing the same behaviour (which, could they speak English, they would associate with the sound “pa-ra-bo-la”) a string of dendrites encoding this characteristic of the universe forms. This is the reality of maths.
Now, the near-miraculous nature of the human brain is such that this is just the first step. These characteristics of the universe are cross-referenced and filed away in an impossibly byzantine structure on which “random” permutations are based, yielding new imaginary characteristics which can be compared to the real universe. With each cycle of observation, memorisation and hypothesisation the “reality” of maths-as-chains-of-atoms becomes evermore distant.  Nevertheless, that reality, that physicality, is to me all that can be said to exist.
A “language of certainty”? Such a phrase does, perhaps, have some resonance with my description of what maths is. But as with many of the debates we have enjoined here, erl, I find that when I go all “first principles” on you like this, I struggle to keep up with the specific aspect of the OP you wish to explore. Apologies if I’m railroading you once again!