I like the way you phrased that.
By placing the “not” where you did, you point out it’s perfectly possible to entertain mistresses at home. As long as you’re sufficiently non-obvious = discreet. In a big enough estate house that might be pretty easy really. 
Which presumes a level of literacy or a higher class of customer than the typically tavern. But surely there were taverns where they offered “chicken or fish”? (or did the airlines invent that?) Or at least “I can give you the one-penny stew, or the one-and-ha’-penny with beef”?
was NYC bigger than Philly or Boston in 1780s?
No, NYC didn’t pass Philly until around 1820.
Shortly after the invention of lawyers.
Indeed, there are places even today where there is no written menu. This past summer, we traveled through the Greek islands, and in a few of the older tavernas, we were invited to peek in the kitchen where the host opened each of the half-dozen stew pots one at a time, saying, today we have this, and this, and this, and this, so what would you like? I would disagree that it’s about literacy, necessarily, certainly not at this point; it’s a combination of longstanding tradition and the convenience for the host of not having to write and circulate a physical menu (not even the French style, where there are two or three small chalkboard menus prepared each day which are carried from table to table by the server and held for diner perusal).
Literacy rates were around 20% in the Middle Ages overall, but the kind of people who were frequenting full-on taverns and inns, as opposed to just getting their small beer from a local alewife, would have skewed that very much towards the more literate side - look at Chaucer’s Pilgrims, for instance.
Wikipedia says, “The original version was widely recognized as the United States’ first fine dining restaurant.” So not the first restaurant, just the first fancy one.
His videos on restaurant menus are quite fascinating too.
There’s a Somali restaurant in Cleveland that I think my mother and I might have been the first white customers at. They had no menus, because most of their clientele was regulars who already knew what they served. We just walked in and said “Make us something tasty”, and they did.
There’s this… a tavern that served food…
At this point I can’t help but recommend the movie Delicious. Set in pre-revolutionary France, it tells the story of what is claimed to be the first restaurant. In French with subtitles, it is a lovely movie.
Sounds like Kifayah’s kitchen. They will be missed. They did have menus though. Started with a whiteboard on the counter and eventually a small menu, with one half in English and the other half in Arabic. if you were so inclined, or adventurous, they even sold camel milk!
Which revelation led me to this blog post that adventurous eaters might enjoy
Aw, man, I hadn’t realized they’d closed-- That’s a shame. It always seemed busy. I guess they hadn’t been around long enough to build up a rainy-day fund to survive covid.