It is if there is a second comma delineating the end of the phrase which could be omitted.
How about ditching the “and” entirely? “Not only was I one of the youngest participants in the seminar, I was also one of only three Americans invited to participate.” It’s a bit less tedious, though I’d recommend finding another way to say either participate (attend?) or participants as well rather than use different forms of the same rootword twice in one sentence.
Hell, since we’ve suggested just about every other variation, let’s go ahead and add “bulleted list” format for the sake of completeness 
ETA: For the sake of doing due diligence for the prose, anybody reviewing the sentence structure really should take a look at it in the context where it’s being used. After all, “structurally correct” doesn’t necessarily equal “good”.
Besides, Americans don’t speak the Queen’s English anyway, so the entire question is moot. 
That’s “mute”, if common usage is any indication.