Charles C. Mann’s 1491: New Revelations of the Americas Before Columbus is a work of popular history, so it must be taken with caution.
However, he has an entire section titled Numbers from Nowhere which recounts the battle between those scholars who have argued for lower population figures pre-contact and those newer scholars who claim much higher numbers. The book is well-sourced and contains an extensive bibliography.
His conclusion is never quite spelled out. Reading between the lines, he seems to come down for a middle ground, giving compelling reasons to doubt the older scholarship which kept shrinking the population estimates by nearly 80% over the first part of the last century, but not quite buying into the highest numbers tossed around.
It’s important to note, however, that the high end, the ones that result in the 97% drop, place that drop over a full century. There is a graph of the research by S. F. Cook and W. W. Borah of the decline of their Aztecs, taken from a variety of contemporary sources.
The graph begins in 1518 with a population of 25 million. Three separate smallpox epidemics dropped the population to about 5 million in 1545. Then came cocoliziti (identified here as Hemorrhagic Fever) and the plague (or a set of diseases the Spanish writers lumped together as the plague) followed by more Hemorrhagic Fever, influenza and measles, another Hemorrhagic Fever, another smallpox, and more measles. That left 700,000 in 1623. That’s ten separate killing epidemics in just over a century. A pattern like that could indeed kill off virtually the entire population through the disruption of food growing and gathering, transportation, marriage and breeding, and general trauma.
Also important to note that such a pattern is not deliberate on the part of the Spanish, but an unintended consequence of their long-term immunity to such diseases, being the descendents of survivors of them.
For a cross-check from a wholly different perspective I looked at A Concise History of World Population, by Massimo Livi-Bacci, an Italian population scientist.
He also doubts the highest numbers and cites Cook and Borah, but he mentions others like R. McCaa who stress the number and the variety of the epidemics that probably literally decimated the population in the hundred years after conquest.
If you stop thinking of the population loss as a one-time event and turn it into an ongoing process that lasted generations, then the case for the numbers is made much stronger. Take the starting population down to 7 million, a mid-level figure, still gives a 90% reduction for the Aztecs. Other populations like the Caribbean Islands were certainly equally hurt. It’s hard to find populations which weren’t.
Disagreement among scientists is not unusual, even when heated. It doesn’t devalue all the work. The lack of precision is certainly frustrating, but it helps that scientists from a number of separate disciplines are attacking the problem and narrowing its boundaries. That makes the conspiracy theory claims easier to dismiss as well.