You do recognize that that is choosing your clothing with how others see it in mind? Not dressing for yourself?
I personally am, well, both. Mostly I like to not be too noticed but not completely.
First importance is that I feel comfortable. Physically.
If I am going out with my wife second is for her. She gets veto power. Mostly. But her comments are otherwise ignored.
To work there is a certain costume/uniform element. Fun ties and/or socks are pretty standard for the image I want to send.
I enjoy being a bit different without calling attention. I am, for example, waiting for “new” glasses to come in: an antique pince-nez frame. I had one before and I did enjoy getting compliments about them.