OK, this thread has given me the courage to deal with this, so why wait? I knew there was a bottle of Agway wasp spray under the sink, apparently left by the last tenant. A lot of them died, but I was wrong when I said it was the size of a fist. It was the size of TWO fists. I have no idea if I got them all. The spray went everywhere, since the gap between the windows was really small, and I had to stick my whole arm up there to get any of it on the nest. OTOH, the window prevented any of it from getting in my eyes. Two wasps escaped but I capped 'em. Didn’t get stung and wore thick rubber gloves, so that’s OK. I’ll check it out tomorrow during the day to see if they’re still there.
Thanks for talking me through this. It was kinda scary and awful, but ultimately it worked out, I hope.