There are some really nasty chemicals that go into the production of semiconductors, but it’s possible to build factories in such a way that you keep all of the nasty stuff under control and don’t just dump it in the nearby river. In fact, EPA regulations have forced the folks who make semiconductors and other electronic things like PCBs and components to really clean up their act in the past few decades. Of course, making a clean factoy costs more than making a dirty factory, so gives the folks overseas who don’t have to worry about the EPA an advantage. Ignoring the economic side of things though, yeah, it’s quite possible to mass produce solar cells and not completely muck up the environment.
It’s the economic side of things that gets you though. Even when solar cells are manufactured in places where they don’t much care if you toss the nasty stuff in the river, the solar cells aren’t cheap enough to compete with other forms of power generation.
Solar cells in the 70’s were ungodly expensive. These days the cost is down to the point where it’s at least in the same ballpark as other forms of energy. There are some folks (mostly rich folks like movie stars) who have converted their houses to solar energy and haven’t gone broke doing it.
Another problem with solar that you didn’t mention is the fact that it doesn’t work so good when the sun isn’t out. This probably isn’t much of a problem for california, but the folks up in washington state are going to be ticked off when they’ve had a week straight of rain and all of their batteries are dead. If the price of oil keeps shooting up, you may see a lot more solar power being used in the next couple of decades, but you’re not going to see it universally accepted througout the US just because of the sunlight issue.