Actually, that last part isn’t right. The number of all integers is the same as the number of all even integers. As a proof: If you can make a one-to-one and onto function between two functions, they have the same size. That makes sense. If you have one set of {A, B, C} and another set of {D, E, F}, you can make a function f(A) = D, f(B) = E, and f(C) = F. So, you have one set, X, of all integers and another set, Y, of all even integers. For every y in Y there is an x such that, y = 2*x.
(Sorry, that’s one of the few proofs I know with all the proper terminology and stuff, so I like to throw it out there)
However, you are right in your initial statement, there are different sizes of infinities. For example, the number of all real numbers is different than the number of all integers.
But if time is infinite, then there was no start in the first place. We wouldn’t have started at -∞ and counted to 0. We would not have started at all.
Here are two more arguments.
Suppose we have two number lines. One is the standard number line, which is infinite. The other is truncated so it only has the numbers -10 to 10. For both, we ask, is it possible to put a dot on a number infinitely to the right of zero? In both cases, the answer is no. Since one is finite, and the other is infinite, clearly the answer to that question gives no indication as to whether the whole thing is finite or infinite.
Now, suppose a magically perfect clock. It’s perfectly accurate, never wears down, and isn’t cyclical. It just displays a number. Right now, that number is N. One second ago, it displayed (N-1). A second before that it was (N-2). A billion seconds ago it was (N-10^9). A googol seconds ago it displayed (N-10^100). Obviously, it has never displayed -∞. But that doesn’t necessarily mean that there is an X such that the clock displayed (N-X), but not (N-X-1). Remember, saying “Well, it had to start at some time”, is assuming that it started.