You’ve never worn panty hose and heels, obviously. But, like a tux, a good pair of shoes and proper fitting hose or stockings goes a long way.
My family went on a Disney cruise in September (it was awesome, BTW). I brought a pair of dockers that served as my “dinner” trousers every night. I brought a blazer, but I didn’t wear it much. We skipped the formal thingie, but I understood that some people wore suits, not tuxes. Other than dinner (and the show we dropped in on just before dinner), it was shorts and bathing suits the whole cruise.
And there were people who ignored the dress recommendations in the dining rooms, and they weren’t thrown overboard. If you’re the type who doesn’t care about flouting convention and being the “oddly dressed” person in the room, I don’t think anyone on the staff is going to mention it.
I get the formal night. It’s a nice change of pace and, of course, completely optional. If it ain’t your thang, then just skip it. If it’s like most cruises, you’ll not starve if you skipped every single dinner. It’s a sin how much food they serve, IMO.
Now what I would definitely feel bitchy about is that you aren’t allowed to wear shorts to dinner, ever. That’s just stupid. You’re on a Caribbean cruise and half the people are walking around in revealing bathing suits all stinking day. Who really cares if you bare a little thigh during dinner? Maybe if it were a hoity toity cruise line, but Disney = informality, IMO. I mean, heck, at DisneyWorld I dined with Cinderella in her freaking castle wearing shorts and a tube top. No one batted an eye.
Okay, I’m kidding about the tube top. But you get the picture.
I’ve been on several cruises and there were always a few on formal night that were dressed in their usual jeans and polo shirt. So wear what you want, the cruise line will still let you have dinner with the more dressed up diners.
How your wife reacts to your choice of dinner wear is another story. 
On the cruises I’ve been on, Formal Night and Semi-Formal Night are also Picture Nights. The one time you can get Mrs Damn, Mr Damn and the whole Damn family clean and shiny and get a good photo.
