Genetic relatedness decreases with the inverse square law, and even first cousin marriages were quite common in the US before 1900.
While I am definitely of the modern mindset that cousin marriage is wrong, the risk of a 40 year old women having a child with a genetic disorder is higher than first cousins.
Remember the risk is with recessive genetic disorders. The minimum viable population is really about risk, and with ~100-200 individuals you could, with management, repopulate the world. Lower numbers are higher risk, and at higher risk of divergence genetically but it is possible.
Also remember that in some time in the past 100,000 years there was only about 1000 humans or less worldwide for a very long time.
Now with easy movement and communication the risks aren’t worth it but we actually have real world examples with other species rebuilding with 20 individuals without controlled breeding programs. But the risk of going extinct due to a recessive genetic trait go way up with numbers that small.
Had the original settlers of Easter Island arrived with a large number of significant recessive genetic traits they would probably have gone extinct too.
Note that even at the time of Charlemagne to someone who is directly related today, the modern person will have 4,000,000,000,000 direct great-grandparents at that level, which is about 70,000% more individuals than existed in Europe at the time. So the very minor differences that are made up among all populations in Europe is mostly due to inbreeding.