It sounds to me like the debate is coming down to whether or not self-replicating probes are possible and something that might be done by an advanced civilization. No SRs, no Fermi Paradox. Is that about it?
In that case, we should just discuss self-replicating probes and,whether or not they are possible.
If they are flying to other stars in decades, then they are traveling at significant percentages of the speed of light. That brings back the energy problem.
Accelerating 1 kg to 1/3 C with perfect efficiency would require 1,205,451 MWh. Double it to slow down at the destination. How many millions of kilograms would a self-replicating probe have to be? And due to the rocket equation, the probe will have to be overwhelmingly fuel by weight, so however large the probe is, it will be a tiny fraction of the weight of the entire system.
If you have the ability to create and manipulate matter and energy like that, why colonize other stars?
And why do you think it can replicate itself with an asteroid? Something as complex as a self-replicating probe probably needs a huge amount of different elements, all of which are unlikely to be found on a single asteroid, or perhaps on any asteroids.
How does the probe know where to go next? Does it have to build its own telescope and search for its next destination? And survey its new system to find resources?
It seems far more likely to me that a civilization that had the ability to send complex probes to other star systems would simply set up a probe factory and send them all out from the home system. Then you don’t get exponential expansion, but you maximize the speed at which star systems could be probed in the short term, which seems more important.
How much failure rate can you withstand? If every probe fails 50% of the time or more, you get probes that maybe make it to a couple of star systems before failing, and you never get geometreic expansion.
It seems to me that we are doing a lot of assuming that somewhere, somehow a civilization would build self-replicating probes that work and colonize the galaxy. I don’t think our understanding of everything from life to information theory is strong enough to make that claim. And the evidence we have so far is that it didn’t happen.
My priors are such that if we can’t see alien self-replicating probes everywhere, it’s more likely that they were never developed for whatever reason than that we are alone in the universe. The latter seems far less likely to me, and it violates the Copernican principle. So the null hypothesis should be that there are likely alien civilizations out there, and they never developed self-replicating probes and deployed them. Or, they tried and the failure rate was such that the expansion collapsed after a time.