Should you care what other people think of you?

Yes.

If you want to be a productive member of society, you need to have people think you are personable and honest. You might get by without the personable part, but you will never succeed without honor and integrity.