Don’t do that. Everyone will assume you never clean your clothes.
I’m about as fashion-challenged as they come. (Fashion-blind is morelike it).
I wear suit-quality dress slacks & a suit-appropriate shirt without tie every day. Here’s what I do:
Buy 6 pants; 3 in shades of black/gray/blue and 3 in shades of tan/brown. Buy 2 pairs of identical dress shoes; one black, one brown.
Buy 5-8 shirts that go with black/blue & 5-8 more that go with tan/brown. Depends on how often you like to do laundry. Make the color distinctions obvious. No off-whites that are sorta tan. For the blacks I have white, blue, gray, red. For the browns I have tan, brown, beige, yellow, purple.
I now have two complete sets of clothes, which I think of as brown & not-brown. Because I have two colors & the workweek has 5 days, it’s easy to get an alternating schedule going that mixes everything up but takes zero thinking. I wear pants a couple times between cleanings, but use a fresh shirt each day.
How’s it work?
Rule 1: Between brown & not-brown, what color do I wear today? Whatever color I didn’t wear yesterday. How can I remember what that was? Look at the top shirt in the dirty clothes pile. Whichever pants I pick, I use the next shirt which goes with that color & the right color shoes.
Rule 2: Is today Wed or Thu? If so, get the next fresh pants of the correct color (see above) and put the old pants in the dry cleaning pile. Otherwise, wear the pre-worn pants of the correct color.
Rule 3: After stuff gets cleaned, hang it up in FIFO order. Always put clean stuff on the left end of the rod & take the next clean item from the right end (or vice versa).
So each week I wear last-week’s pants on Mon & Tue, then fresh ones on Wed & Thu, re-using Wed’s on Fri. Alternating between not-brown & brown.
The result is in any workweek people see me wear 4 different pants, alternating colors every day. Pants get worn 2x or 3x each week. I only need to dry clean 2 pair a week & have 2 weeks to get then in & out of the cleaners. And with a bunch of shirts not divisible by 6, the shirts & pants mix themselves up with no thought. It just works.
Sounds kinda elaborate or OCD to describe, but after 20 years of wearing a uniform to work I needed a method which took no thought to execute. Once I came up with this pattern, life is easy. Just grab the item on top & put it on. Avoid dumb combos like black pants & a brown shirt or shoes. Even I can handle that.
The key to simplicity in operation is to avoid buying borderline-color clothes in the first place. Make sure everything is either obviously brown or obviously not-brown.