Look, there is stuff I don’t tell my wife, but if I’m being honest, it’s for selfish reasons, not to protect her.
If your wife knew your friend jerked off with her panties, she would not want him in the house. It’s her house too, right? She has as much right as you do about who can and can’t come in the house?
Wonder if he’s been using your boxer shorts the other times you’ve not been around to catch him?
OK, some time has past so I can relate this story.
One of my exes, she had a little brother. His is the only boy child and the youngest of 4. Once, his mom, found in his room three pairs of panties, one from each of his older sisters. He had been masturbating in them. They thought he wanted the silky texture and that was it.
Yeah, mmmm, that’s it.
But you know, that was years ago and everything is fine now.
I could forgive my husband not telling me. I could not forgive my husband allowing this person into our home again, and I would be annoyed if he didn’t tell me that the friendship was over.
If the friendship weren’t over? Well, I know the man I’m married to. The friendship would be over.
Lies of omission aren’t great in marriages either, Crime Scene, which is what you’ll be doing by withholding this information from your wife.
Does it interest you at all that the female Dopers have almost uniformly said that you need to tell her? This isn’t, “My friend cheated at golf, and my wife doesn’t need to know.” This is, “My friend is truly creepy and has problems with boundaries, and probably used a pair of your panties for masturbation. I understand if you want me to never hang out with this loser again.”
My reaction too.
If my husband had walked in on this scene when he was still alive, he would waste no time in telling me and we’d both probably be laughing our asses off. We wouldn’t have cut the friend off, but we’d have difficulty keeping a straight face around him and we’d probably develop a bunch of inside jokes about getting a lock for the underwear drawer and such things.
I mean, it’s pervy and the guy got caught, but everyone has a perverted side.
PS I used to have a CAT that did this…he would go into the hamper, drag out a used pair of undies, arrange them on his head so his nose was in the crotch and go to town
Closer to my reaction, too. Probably have a good laugh about it, but because of the sum of his creepy behavior, avoid contact until the relationship died a natural death. I certainly wouldn’t feel scared or outraged, just mildly peeved that he pilfered through my things. And of course I’d trash the panties, 'cause eww gross.
We really can’t ever know who’s enjoying a good wank over us. Thank God for that, right?
Torch? Check. Pitchfork? Check.
KILL IT WITH FIRE!
But seriously, throw him away because of this:
He’s not your friend. He tolerates you because it gives him access to certain types of sexual gratification that he cannot otherwise get. The only reason to keep him around is that you derive some pleasure from his company that allows you to tolerate his creepiness.
Women who meet this guy in person think he’s a creep. Women on the internet who hear about him think he’s a creep. You have pretty damn airtight evidence that he’s a creep. Don’t attend his wedding. The whole time he’s going to be thinking about how he really got one over on you, the chump who lets other guys have their way with his wife’s undergarments.
As for telling your wife, please don’t take this as a perosnal attack, because I don’t know y’all from Adam, but… If she is emotionally immature and your relationship is one of paternalistic condescention, maybe it’s a good idea to lie to her about why you don’t want Creepy Mcpantywacker alone in the house, to protect her fragile little feelings. If she’s an adult, entitled to respect equal to yourself, tell her. Maybe she’ll freak out, maybe she’ll laugh her ass off, maybe she’ll make excuses for him, either way, she’s an adult, it’s her house and her underpants.
Also, was he in San Francisco last week? If so, did a lady threaten to stab him while he was innocently sitting on the floor of the elevated train? Just wondering.
I’ve been following this thread all day, and refrained from chiming in on either or any side of the spectrum of choices for how you might proceed. I see merit in a number of directions you could go.
Trust your gut, man.
No part of this discussion should be considered cut-and-dried. It’s just so weird as to be not that simple.
I don’t envy you being in this position, but I’ve a feeling that your instinct on the matter shouldn’t be ignored, whatever it is. You seem to love and respect your wife, so I’m pretty confident that you know what you’re doing.
Best. Username. Ever.
Nobody secretly jacks off in my wife’s panties but me.
Anyway, I like to think I would have thrown him out right away, ended the friendship, and told my wife.
This.
It wasn’t just a pair of panties. It was a pair of panties infused with the scent of your wife’s hoo-hah.
Come to think of it, I probably would have beat the shit out of him before I threw him out, and then told his fiancee, just to be a dick about it.
These things never end in a hot, hot foursome with panties all over the place, do they. 
panty sniffing is pretty common. Tell him to go online and buy some and leave your wife’s dirty drawers alone.
Another option would be to invite him over for a beer, set up a hidden cam and make another run to the store. Then you’d have a funny video to show at the wedding rehearsal dinner.
Stealth brag?
Surely Ben Stiller and Owen Wilson could execute some kind of covert sting operation?
What about Vince Vaughan and Kevin James?
Paul Rudd and Jason Segel?
Heh - you reminded me of a scene from the British tv show, “Misfits.” 
Do you care if he was jacking off with your wife’s panties? You seem less than upset.
Would be almost as good as this one:
I think that this is just about the most perfect response you would ever get.