If you’re talking demi-humans, Dwarves are a good choice. They don’t even have magic users–or they didn’t under 1st Edition + Unearthed Arcana rules, which is what I played/DM’d back in the day. Elves are slightly more problematic, as they are a magical race, and should have some casters…
As for keeping down the magic man–you don’t have to do that by fiat. There are multiple legit combat strategies to defeat/minimize casters. They wear cloth armor and have 4-sided hit dice. Any intelligent opponent is going to make the casters a priority target as soon as a spell is cast. Also, it’s not like EQ/WoW where casters have the ability to cast umpteen uberriffic spells, drink/regain mana and cast another umpteen uberrific spells. A 5th level caster gets what…1 3rd level spell, 2 2nd level spells, and three 1st level spells (going from memory, could be off a little). They’ve got to choose which spells to memorize prior to starting an adventure, and once they’ve cast all their spells, they’re done for the day as far as magic is concerned…unless they have a wand or a staff or something. To get more spells, the caster must spend hours in undisturbed study…or clerics spend hours in prayer/meditation.
Spells have casting times. If a warrior sees a guy in a pointy hat waving hsi arms and speaking in tongues, the warrior may thraw a dagger or shoot an arrow at the caster…if it connects before the spell is cast, then the spell has no effect, and is erased from memory. Initiative checks before every combat round, also require casters to declare, prior to initiative, whether they are casting anything, if so, what spell, and at what target. Use range, material components, etc to limit the effects per the rules.
The first wizard that unloads a fireball on an army will likely get pin-cushioned by archers…so they can’t just load up on offensive spells. They need to have some defensive abilities as well…in this case “Protection from Normal Missles” would be a good choice. Mirror Image can be useful in some situations…even stuff like Web or Stinking Cloud. I’m a huge fan of Slow.
Also, spells are not easy to get. A first level caster does not get to open his player’s handbook and choose to memorize any spell on the list. He only has access to the spells in his personal spellbook, and at first level, he’s probably only got 3-5 to chose from. Every other spell in the game must be located by the caster…either a higher level NPC is willing to teach the PC caster a particular spell, (for a hefty fee and/or service), or the PC has to find the spell in a scroll, or in a captured spellbook of an enemy caster. Even if a spell is located, the PC may not be able to understand it…there’s a percentage roll based on intelligence to check for that. Once a spell is found and the PC understands it, he’s still got to copy it into his own spellbook…and that might require special ink/materials that are not easy to obtain.
It is quite possible to have a caster, whether PC or NPC, that has reached 7th level and still doesn’t know Fireball…or Lightning Bolt…or whatever spell you think raises a balance issue in your game.
For clerics, their god may not allow the use of certain spells…the PC can pray for those spells if they want, but they won’t have that prayer/request granted. Or they may require their clerics to always have certain spells. For example, I played with a cleric of Apollo. That cleric was required to always have either Light or Continual Light as one of his spells, because Apollo is a Sun god. Cleric was also required to carry at least one healing spell at each level a healing spell is available–because his god was also a healer.