She is only slightly bothered by them, nowhere near as much as me.
Yep, this. Exactly. Or even worse, my kids will somehow end up amusing/playing with/looking after the young’uns.
Well, they usually take it to another level.
Nahhh, I really want to go. I guess I just wanted some reassurance here. Maybe it won’t be the horror that I envision.
Honestly, I don’t see why it’s so hard to say something like, “Those kids are a nightmare. This vacation will be no fun at all if they come.”
Remember your name, MEAN Mr Mustard.
Don’t you worry your pretty little head about it. It will be nowhere near the horror you envision. It is sure to be far, far worse! 
My kids are currently 18-22, and I hope my attitude towards little kids changes before they start popping out grandbabies, because right now my tolerance for even well behaved kids is mighty thin.
Irrelevant now that I have read the rest of the thread.
I may just have a way overactive imagination, but this is always the case for me. I can imagine (and get quite worked up) and then it ends up being a non-issue/non-event.
If you value the time with your friends as much as it sounds like you do, I think you are stuck for this year. But maybe towards the end of the vacation, when it’s just the 4 of you around, you can start a conversation along the lines of:
“Wow, having kids around tires me out a whole lot more then it used to. I don’t know how my grandparents used to put up with all us grandkids. Maybe next year we can arrange things so that half the time the kids around around, and half the time it is just us old coots.”
I think you’re stuck going with the kids this year so it’s time to adjust your attitude a little and buy some sound canceling headphones.
Really, they’re not your kids or your responsibility so when they start acting up you can read, have a couple more drinks or do something else with your wife. Your wife probably isn’t as bothered because most moms have developed a talent for ignoring other people’s annoying children (which are usually all except your own).
I imagine your friends probably think their grandchildren are kind of a pain too but are too wonderful to mention it.
Let them know that you are looking forward to this vacation and that you are always glad to see your friends. Mention that the kids are full of sunshine and a delight to be around. It sounds like it’s already understood that from time to time the two of you will be going off and doing your own thing, so you probably won’t have to voice that point. Then while you’re over there just conveniently happen to have arranged to spend more time by yourselves or with other adults than around the kids. 
Errrrrrr…I believe we have arrived at the solution. 
Yes indeed. They give off cues to this effect from time to time.
We have a somewhat similar situation - my family has started having a weekend in the mountains every summer, which would be lovely except: 1. My sister’s two kids are big pains in the ass, and 2. It’s just a big, drunken piss-up all weekend, and I don’t drink. If I want to see my family and spend some time in the mountains, I pretty much have to tolerate these annoyances. There really doesn’t seem to be a way to make compromises when you’re dealing with a group and you’re the odd man out who doesn’t like things the way everyone else likes them. You can try, but I don’t hold out much hope for you. Sorry. 
I agree that you’re only hurting yourself if you spend your vacation in a situation you don’t enjoy, simply because you’re afraid to say anything.
If it were me, I wouldn’t communicate this by note or e-mail (my generally preferred method for discussing uncomfortable subjects), but instead communicate that directly by phone with the person doing the planning. The sentence quoted, followed by “That wasn’t really possible with all the kids around. I appreciate the invite, but I think we’ll plan on something a little…quieter…for this year. Thanks.”
I would simply suggest that you would like to have, as you haven’t in years, separate vacation, you know, to be more romantic, etc. Then book it.
It’s your vacation.
Maybe they need to have a “kids table” version of the vacation… there, but very far away, and out of shouting distance.
I’m not ‘afraid’ to say anything; I’m just weighing whether it would wise to (since it would make everyone, including me, unhappy).
Well, it’s not really like that. If anything, they’re tagging along on our vacation. It’s been this way for 20+ years. And again, I want to vacation with them, which is why I’m leaning toward just toughing it out.
On the one hand, the longer you let it go without saying anything, the more everyone will think you’re okay with it. On the other hand, this is a problem that will take care of itself in time as the kids age, if you can wait that long. How about approaching your friends with a compromise in hand - every other year they can invite all the kids they want, but you get one year in between with just them? And maybe that’s how you can say it - “We love spending time with the two of you, and the kids really take up your time.”
Is it possible to change the venue, so there’s not room for the extra people?
Where are you going?
Yes I think the details of the vacation will affect what you should do. If it’s your beach house that you own, then you can disinvite the family with kids, although you are right that the polite time to do that would have been when they first brought it up. You would have to hope that old friends understand why you are just mentioning it now. If it’s a campground where everyone gets their own cabin then really all you can do is tell them you’re not going this year and see if they ask why.