It’s only the larvae (caterpillars) of clothes moths that do damage, as they are the only feeding form (adults do not feed). However, the adults obviously have to find the clothes first so they can lay their eggs there. This is what Doug was talking about.
Clothes moth larvae mostly feed on clothes made of animal hair (wool and fur), not vegetable fibers such as cotton (although they may eat cotton/wool blends, and use cotton or other fabrics for their cocoons, and one of the two main species may sometimes eat tobacco or hemp.)
In nature clothes moth larvae feed on hair, feathers, wool, fur, dead insects, dried animal remains, and pollen. I believe one typical habitat is animal nests lined with fur or feathers.