Well, if you can move through space then you’ve just invented really fast space travel to just about anywhere in the universe. In most fictional settings that’s not part of the time travel tech.
Well, there aren’t any fixed points in space that you can measure, so as far as I am aware there’s no way to establish your absolute position in the universe. But if you take the 2-D equivalent of the universe (just to make it a little easier to grasp the concepts), you’re not traveling around on the surface of a sheet of paper like Flatland. You’re traveling around on the surface of a balloon (which is slowly inflating). If you imagine you are riding in a little car and are traveling around the surface of the balloon, then you magically pluck yourself out of the car, rewind the film to go back a bit, then put yourself in the same place on the balloon, the car isn’t there.
If fictional settings would come up with some sort of even semi-plausible explanation for the xyz shift in addition to the t shift, it wouldn’t annoy me so much. But in most fiction, you don’t have that.
It annoys me. It doesn’t have to annoy you.