Okay, for the sake of completeness, here are a few models of how time might work, some of which allow time travel, others don’t.
1/Presentism. In this model only the present exists, the past has gone, and the future does not exist yet (and when it does exist, it will be the present). I suspect that this is the model that msmith537 subscribes to in the OP, either consciously or otherwise.
More details here
In Presentism there would be no past or future to travel to.
But there are other models.
2/Eternalism. In this model the past and future both exist forever, outside of time as we know it, and we pass through time in a similar way to the way we move though space. So we could in theory travel back to the past, which stiull exists elsewhere in space time, and experience the same events again. In some models of time, the entire universe, past present and future, exists as a solid four-dimensional block, through which we move like a three-dimensional slice, always moving in one direction. This is the block universe concept, and if both [past and future exist as a single, unchanging block, then there is perfect predestination and events are fixed. Some might say that in a block universe there is no free will, but this is a completely different question.
More details here
In Eternalism there is a past to go to, but if the universe is a solid four-dimensional block we can’t change it withouit completely destroying the existing pattern.
Another variant of the block universe concept is the ‘growing block universe’ concept, where only the past exists as a solid block, but the future does not exist yet. The growing block universe looks like it might allow time travel at first glance, since the past exists and can be travelled to; but if the future does not exist, then there would be no future to travel from.
Another different view of time is the multiple timelines possibility, which means you could travel to the past and the future, but they need not be the past or future you are familiar with. I think that, if expressed in ‘block universe’ terms, this implies that the universe has (at least) another dimension, one in which events can vary significantly. Multiple timelines are popular with science fiction authors, but I think they are less popular with physicists and cosmologists.