In fact this is known as the cosmological twin paradox.
Take for example the Einstein static Universe, which has the geometry and topology of hypersphere and space is static (neither expanding or contracting). It is perfectly possible for one observer to stay at home on his planet (which we assume doesn’t move relative to the fixed background of stars/galaxies), but for another observer to circumnavigate the Universe, without accelerating and starting and finishing at the planet. In this case the observer who circumnavigates the Universe experiences less time than the observer who stays at home. So as Chronos says there is a preferred reference frame decided by the fixed background of stars/galaxies.
In fact even if space is completely empty and has a flat geometry and you can still set up a similar situation. Take flat Minkowski space from special relativity described by an arbitrary set of (Minkwoski) inertial coordinates (x,y,z,t), if you make the identification (a,y,z,t) = (0,y,z,t) for some value of x, a, greater than 0, then you can “roll up” space into a sort of “hypercylinder” , much like you can roll up a flat newspaper into a tube without creasing it. In this situation an observer who remains in the inertial frame which was used to create the “hypercylinder” experiences more time than an observer who travels around it.
The reason is that non-accelerating observers in spacetime have timelike geodesic wordlines. In many spacetimes(like normal “non-rolled up” Minkwoski space), timelike geodesics can only intersect once, but others (like the above two examples) they can intersect more than once. The definition of a timelike geodesic is that it locally maximizes the proper time along it, but that doesn’t mean it globally maximizes the proper time, which is why one inertial observer can experience more time than another inertial observer, even when they start and finish at the same time and place.
In Riemannian space (as opposed to Lorentizan spacetime) a geodesic locally minimizes the distance along it, so an analogy would be two travellers who want to go from New York to Washington DC. One traveller sets out in a South-West direction and travels 200 miles, the other one sets out in a North-East direction, but has to travel over 24,000 miles. They both take geodesic (locally distance minimizing) paths on the spherical surface of the Earth, which start and end at the same point, but one is significantly shorter than the other.