The reason why is pretty simple: In IV-VI, droids are used as information processors (C-3PO, R2D2), medical droids, and in a few other pretty limited capacities. Droid technology seems fairly expensive. They are everywhere, but at the same time a family like Luke’s doesn’t have a huge passel of droid servants. Jawas scavenge droids and droid parts and trade them because it’s worth the effort (i.e., droids are valuable).
Further, droids are rarely warriors, and when they are, it’s special. IG-88 the bounty hunter droid is one that comes to mind.
In I-III, Lucas of course lays on the stupid in every way possible, and the massive droid armies are one big example. But there are plenty others. There are the destroyer droids that are a match for Jedi, and the IG-100 Magnagard droids that are supposed to be fierce fighters but are derpy.
This brain-dead “ecosystem” completely cheapens the value and role of droids simply by the sheer number of them. But they are also stupid and shitty in every way possible. As warriors, they make the mooks going after James Bond look like fuckin’ Achilles. In the 21st century, our drone technology is already better than these shitters. How would a real droid army with strong AI and laser rifles behave? Well, I think the following is plausible:
• First and foremost, they would contain bombs and would not have to get very close to something to blow it the fuck away. This is already 21st drone tech. No finesse required.
• They’d have a bunch of sensors and perfect eyesight and misses with those rifles would be rare.
• They’d be fast as fuck. None of this tottering around like tinker toys. They’d be running around like lightning and annihilating everything.
• They’d have perfect communication with each (no derpy voices required!) and would instantly adapt to anything and everything on the battlefield, swarming like rats or retreating, hiding, and sniping like guerrillas. With their advanced AI, they’d immediately understand the limited battle tactics of meat soldiers and react efficiently.
• They’d use their droid-ness and disposability to their advantage. They have no fear, so they’d use suicide tactics regularly. And why not just douse the battlefield in poison gas? Who needs lungs?
When you remember the prequels (as painful as that may be), do you remember the battle droids doing anything? They are just standing there getting blown away. In reality, they’d shred anything organic in their path. Jedi? Don’t care if you can cause laser bolts to go bouncy-boo off your lightsword. These guys would shower you with fire in a solid sphere around your body, and you’d be dead (in Episode III, Obi-Wan and Anus-kin have to retreat from destroyers, which at last is a bit of realism with respect to this tech).
I know no one is going to want to defend anything about the prequels, but there you go. I think the use of droids is one of the dimmest and dumbest things in those crap-a-ronis.