But your argument, to work, needs an infinite amount of moments to have elapsed. It’s a case where the finite and the infinite part ways, and what works for the finite case no longer works in the infinite case. If there were, say, twenty moments in the past, then twenty moments would have elapsed since the beginning. The two notions are the same, for the finite case.
No longer in the infinite case. There, even though there is an infinite number of moments to the past, in time, that is, since any given moment in the past, only a finite amount of moments have elapsed. It’s similar to Hilbert’s Hotel: in the finite case, if you have twenty rooms, and twenty rooms are occupied, no more guests will fit. But again, in the infinite case, the two notions are no longer the same: even though infinitely many rooms are occupied, you can always make room for more guests.
Really, you can confirm this for yourself: find two numbers such that their difference is infinity. If you can, then your argument works; but if you can’t, then this means directly that there is no infinite amount of time between any two moments, and thus, no matter ‘when’ you are, there is no moment such that from then, it would have taken an infinite amount of moments to get to now.
Again, consider the finite case: twenty moments to the past, so from then to now, twenty moments have elapsed. But for the infinite case, this simply ceases to be true: ‘amount of time to the past’ and ‘time elapsed until now’ cease to mean the same thing, just as ‘number of guests in hotel’ and ‘number of rooms occupied’ cease to mean the same thing in Hilbert’s Hotel.
This isn’t intuitive, but the math is easy: subtract any two numbers, and your result will always be smaller than infinity.
I don’t think it’s a contradiction that a quantum field has infinitely many degrees of freedom, for instance.
Yes, but from none of those moments in the past, it takes an infinite amount of moments to get to here; the distance between any of the infinite amount of moments in the past and now is always finite. Check them one by one: time between now and one moment to the past—one moment. Time between now and two moments to the past—two moments. Time between now and n moments to the past—n moments. Time between now and n+1 moments to the past—n+1 moments. So, by induction, for all moments in the past, the time to get to now is finite. Only if that were not the case would your argument work.
This is just confusing a limit of perception with a limit on what exists. If we could perceive no greater quantities than 500, would that mean that nothing could be in greater quantity than 500?