American here. For us, “underwear” (or “underpants”) is the typically legless garment that covers your genitals and buttocks (more or less) and fits directly against your skin. “Pants” are the long-legged outer garment you wear over your underwear.
“Not wearing pants”, interpreted literally, would mean only that the long-legged garment is absent. That literal interpretation wouldn’t preclude the presence of shorts or a skirt, but if I wanted to communicate to someone that I was wearing shorts or a skirt, I would not tell them “I’m not wearing pants.” To me, that phrase means I am wearing nothing but underwear on my lower half.