The OP is right: Time elapsed depends on your reference frame, and the laws of physics don’t distinguish between different reference frames. However, according to our current best models of the Universe, the Universe does in fact have a set of reference frames associated with it which are, in a global sense, preferred.
This calls for some further elaboration. Here on Earth, I could work in any of a multitude of different reference frames. Any local experiment I ran in any of those reference frames would work out the same way, and give the same results. In any of those reference frames, I could also measure the age of the Universe, and I would get a different answer in different reference frames. This is not a contradiction, because a measurement of the age of the Universe is inherently not a local measurement. In no reference frame would I find an age greater than 13.798 billion years (to within the error bars), and I would find that age in exactly one reference frame. That one reference frame is therefore, in some sense, special, and also has some other nice properties: For instance, in that frame and that frame alone the Universe will appear to be isotropic (the same in every direction).
If I were in some other galaxy, billions of lightyears away from Earth, I could make the same statements: Local experiments won’t care about reference frames, but cosmic-scale measurements would see differences in different reference frames, and there would be exactly one reference frame where the age of the Universe was a maximum, at 13.798 billion years. However, that reference frame would not be the same reference frame as the one that’s favored at the Earth. Every point in the Universe has a different preferred reference frame associated with that point.
By way of analogy, consider different points on the surface of the Earth. I can draw planes in 3D space anywhere on the Earth, but at any given point, there’s one plane that’s special, because it’s horizontal (i.e., parallel to the surface of the Earth). But the horizontal planes at two different points on the Earth are not parallel to each other.