What grammar or style rule determines if “ess” is an appropriate ending to a word?
I suspect the answer is simply euphony. OED gives no hard and fast rule, other than saying the -ess ending is not used in English for the females of animals, the few exceptions (lioness, etc) coming directly from the French.
The -ess ending is ultimately from the Latin, and the Greek -isse. It gradually replaced the original Old English feminine ending, which was -ster, as in spinster.
And we have it to thank for the funnier answer to “When is a door not a door?” – “When it’s a Negress”. “an egress”