I have no answer for the OP (and am not a mathematician), but infinite series seem to tie well into p-adic analysis so there might be a definite answer there that resolves any question of paradox.
Granted, Google isn’t presenting any results along this line and I doubt that any of us could perform such an analysis, so the point is moot, but perhaps someone will come across this one day and give 'er a go.
The nature of the paradox seems to be not in an answer but in an ill-formed question. “after you have performed step N for all natural numbers N” You can never perform this for all natural numbers. The line of thinking that says you have no balls left in the box assumes you will at some point run out of natural numbers for putting in new balls, and yet continue to take out existing balls. I do not see how that reasoning can be valid. You can gin up this business about halving the time to perform an operation and therefore say that by noon you have processed all natural numbers, but by this logic noon is just a limit and you will never actually reach noon. (IANAM so this is a layman’s argument that doesn’t have the rigor that would be needed to prove this.)
As far as an ever-diminished time needed to perform each step, you would eventually reach the Planck time, where it is debatable whether any further subdivision of time has meaning.
The ‛infinity paradox’ [number of balls in a box grows without bound while every ball is eventually removed] appears in Littlewood’s book (1953), but is likely a bit of folklore that was and still is making the rounds at fashionable dinner parties; compare a different puzzle* that he writes ‛swept Europe a good many years ago and in one way or another has appeared in a number of books.’ The Wiki article does not say anything about the history beyond it having been described by Littlewood, but maybe someone reading this knows something more? Paging especially anyone 90+ years old…
As for the paradox itself, like Hilbert’s hotel, it is supposed to provide food for thought and illustrate some counterintuitive phaenomena when reasoning about the infinite, but there is no actual contradiction or mathematical difficulty, in case anyone was confused. (In particular, all this discussion of how one would go about physically performing an infinite number of steps is beside the point.)
*For the curious: Three ladies, A, B, C in a railway carriage all have dirty faces and are all laughing. It suddenly flashes on A: why doesn’t B realize C is laughing at her ?— Heavens ! I must be laughable.
Was this the origin of the well known logic puzzle, I forget the usual name - often formulated as 100 silent monks who are all perfect logicians, with red or blue dots painted on their foreheads and no mirror…
Apparently: