If you go under Settings in your Contacts, you should be able to set how/who you see them (facebook, gmail, just phone #s).
There’s an option somewhere in the address book to display only people you have numbers for.