I hope when you’re looking at downsides, you’re doing so knowing that many of those listed here are disputed. There’s a great deal of misinformation out there, due in large part to insurance companies, conservative politicians, and a certain “news” channel. There’s also some misinformation on the upsides, but you didn’t ask for that.
You should also consider that “single payer” doesn’t have a uniform definition. Canada’s system, for instance, still relies on private insurers and employer/employee contributions as well as taxes, and doctors are not government employees. (Canada spends 10% of its GNP on health care compared to the US’s 20%.) In the UK, government really does run all healthcare, and there’s no charge to consumers except for OTC meds. In Iceland, the government pays 80-85% of healthcare costs; the rest is paid mainly through service fees. In Finland, about 2.5% of taxable income goes to healthcare coverage: 1% by employer, 1/5% by employee.
One argument I hear frequently is that people have to wait for an appointment for months, especially to see specialists. This one always makes me chuckle, albeit bitterly. In some areas I’ve lived, specialists schedule 4 months out, and primary care docs schedule 3 months out. If you have to be seen sooner, you go to a walk-in clinic.
An argument I have is that we’ll need to increase the supply of medical professionals. There’s a serious shortage of PCP’s, and one reason is that the Affordable Care Act meant more people could afford to get health care. Only 2 or 3 PCP’s in my city are taking new patients. The more people covered, the more docs and nurses we’ll need.