Like some pants swish when you walk, while others don’t, and they might both be 100% cotton. Is it the weave, or what?
Eh, it probably has to do with how baggy they are.
Yes, it can be the weave causes this. Surface texture and tightness of weave can mean more noise when fabric contacts another portion of fabric. It can also be sizing or other additves to the fabric like starch that makes the fabric make more noise. Silk if treated with mettalic salts has a characteristic rustle, while un salted silk will not rustle. Taffeta is traditionally noisy and it has a plain but tight weave.
I wore corduroy pants for eight years straight as part of a school uniform and learned to run bow-legged when sneaking up on someone so as not to reveal myself from 20 feet away. “Zip, Zip, zip, zip…”