Dress pants:
Read the label & check the composition of the fabric. You want to kill the mold, remove the stain and potentially any musty smell. None of this is difficult or harmful to the fabric (or you!) but knowing what the fabric is helps. Vinegar is your best bet no matter the fabric composition, I’ve used it to safely remove mold from silk, even. If your fabric is wool, remember that the combination of heat and friction will felt/shrink the fabric, separately they’re ok. So you can also use a steamer.
Do not (ever!) use fabric conditioner. It leaves a layer on clothes and just gives mold something to eat. It only harms clothes — all clothes — and serves no purpose.
I would: hand wash once to remove surface mold, vinegar soak hot, NO AGITATION and 20 min is plenty, steam to kill the last little bit, then normal gentle cycle in the machine.
If there is no wool (or silk) you can use an enzymatic cleaner to soak. Enzymes are amazing!
Closet:
Everything out, hoover, clean with a mold killing product and dry thoroughly, then put a few of those dessicating thingies in.
Washing machine:
We have front loaders in Europe so this might be different (I had top loaders in Brazil but it’s long ago and I can’t remember how they work). Pull out the tray for the powders and fabric conditioner, wash in the sink. Stop using fabric conditioner. Check the filter & read the instructions because I just found out mine has an extra, secret filter (wtf Samsung?). Run a soapy cloth around the rubber crevices where it closes, gunk can hide there. Run the cleaning programme (mine tells me when to run it, frankly, being Samsung, mine is just bossy and talkative in general).
I learned from a consumers guide that with washing machines, you don’t want to buy the most expensive but the middle tier and then replace every 7 years because the new ones are more energy efficient and it makes it worth it. So consider replacing your old one or look into if the same is true for top loaders.
You’ll be fine, mold is easy to treat. Just think of it as science: you need to kill a living thing and then do chemistry to get rid of the stain/smell. It makes it more fun.