After some searching, I found that apparently, there is a type of sweatpants called “drop crotch sweatpants” that have been trendy the last couple years. They look like a tamer version of the harem pants from the '80s. They are meant to be worn low-waisted for the MC Hammer/I-am-wearing-a-diaper look. I am guessing that the bigger brands were inspired by this trend.
Pull them up over your beer gut, fasten. Presto!
Wear them up to your armpits, old-people style.
I was thinking that possibly my lack of gut was my undoing. I considered that maybe people could not relate to the issue because when you have a 40" waist there is no extra fabric on your size small sweatpants. 
This may be a serious answer: these garments may be meant for people whose proportions skew large, and who find the excess fabric makes them fit comfortably, not “baggy” as you describe them.
Buy a new pair a size or two smaller. There’s a difference between “roomy” (good, when dealing with sweatpants) and “baggy to the point of discomfort”. Too many people wear clothes that don’t fit them then complain about the fit.
With suspenders and a belt!
Tuck the cuffs into your tube socks, and loop the waist’s drawstring over the back of your head to the back of your neck like a halter. Simple, huh.
When you are already at small, there is none smaller!
Seriously, I am tall and average weight, but svelte. I am already picking the smallest size available that does not look like I am wearing capri pants. If you look at the photos I linked to of the “drop crotch” sweatpants, you can see that it is a problem of design. Even when the waist fits as it should, the crotch is low. It seems that some people just like that look.
Well, duh. And always with an onion.
Spurring on, not spurning on.