A lot of these insurgent groups have/had power because the United States presence in Iraq was doing nothing whatsoever to make life safer for ordinary Iraqis.
Prior to Petraeus a lot of our focus was on keeping our units bottled up in well-fortified areas and sending them out periodically on “missions.”
The problem is, that makes it seem like we’re not really interested in what is happening to the ordinary Iraqis (and maybe Bush and the military brass wasn’t worried about that), when you just come out of your fortresses to kill some insurgents and then roll back into them–leaving the Iraqis in the street unprotected.
This is why Moqtada al-Sadr’s Mahdi militia became so powerful. The Mahdi militia was basically operating as a state within a state. They were providing the security we weren’t providing. Many of the people who were effectively under Sadr’s rule didn’t like his harsh Islamic law and the many death penalties his “legal system” handed out. But, their choice was between him and us. He was providing them with street-level protection and we weren’t really trying to protect them at all.
Now that we have been aggressively policing the population instead of just trying to chase insurgents all over the country side, Sadr is broken. Just recently he announced that he has failed to establish an Islamic state in Iraq. This is a guy who, in 2006, was arguably the most powerful warlord in Iraq, one who came very close to establishing his own state.
Where this brings paying the former insurgent groups off is, while they are fighting al-Qaeda we are providing more security presence to the Iraqis than ever before. So assuming at some point al-Qaeda in Iraq is truly gone and broken, when these militia groups come back, the Iraqis (ideally) won’t turn to them for protection because they will no longer need protection.
This may lead to some of these insurgent groups attempting to take arms against the state even more–but again, if Iraqis view them as the reason for the security problem they will turn on them. Without the support of a local population it is almost impossible for an insurgency to sustain itself.
Look at really effective terrorist/insurgent/revolutionary (whatever term fits) groups, the IRA, ETA, the Mahdi militia (prior to recently), they were effective because they had a large population they could hide in and that was willing to protect them.
Now, look at the Ku Klux Klan. At one point it was very effective, it had strong support throughout the Southern states from southerners and their governments. Local police departments wouldn’t seriously try to investigate their crimes–if they did, juries wouldn’t convict their members.
The KKK is more or less broken as an organization now because pretty much any jury of southerners would probably convict them. They no longer have the support of the population and cannot survive.
This is like many of the militia/survivalist movements in the United States, they’ve never gotten much traction because they only have the support of a very small fringe–without any support from the population at large they are easily monitored, caught, and kept under wraps.
