I ride about 5,000 miles a year. I also walk about 500 miles a year much of that on sidewalks.
The first problem is that there aren’t sidewalks in many places that I want to go.
Next many sidewalks are in horrible condition. While potholes on roads are bad, some of the gaps, uneven slabs, and other issues make sidewalks hard to ride on short of a fat tire mountain bike. These are manageable while walking at 3 mph but an issue at 15 mph on the bike.
Then there’s obstacles on or near sidewalks such as telephone poles and garbage cans.
Next comes intersections. If you are one a sidewalk, you need to stop at every street before crossing. If there’s a building, you may have trouble seeing traffic coming. While on the street you can just follow the flow of traffic like a car.
I’ve been hit once by a car in 40+ years of riding. It was by a car that pulled out from an intersection. Had I been on the sidewalk, I doubt that it would have made a difference.