I don’t think that’s true, or else calculators wouldn’t be solar powered.
It’s what you first said: Charging in direct sunlight can take a long time, and if you’re indoors or its cloudy, you might as well forget it. Besides, I usually have my phone in my pocket, so it wouldn’t get charged anyway while I’m not using it. However, when I’m at my desk or in my car, I can easily recharge it and its still rather handy. Or, I can plug it in and charge it over night while I’m asleep.
It’s the same reason why we don’t drive electric cars with solar cells on top. They take too much power and can’t recharge quick enough with solar cells.
In fact, a Better Place has plans to have electric car battery switching stations rather than depending upon battery charging. You charge your battery when you’re stopped for long term, but if you’re driving along the highway, you can switch out your battery pack for a charged one in under four minutes.