You are attempting to draw a distinction between a bar that is open to the public and a home that is not. Why not make a distinction between a wooded estate and a bank vault. If you commit a trespass onto an unfenced wooded estate, probably no big deal. The estate owner brings you to court for the trespass. You explain it was inadvertent and caused no harm. A wise judge throws the case out. Now. You commit a trespass into a public bank’s vault and are bopped on the head. You bring suit because you were assaulted in a “business” open to the public. The wise judge throws this out because the bopper was acting within his rights.
The point I am making is that you are being a bit weird about this. Libertarians believe there should be courts. (Some even believe there would be better courts in the absence of the state). A judge would decide in a libertarian society if a trespass was committed. He would take things such as the nature of the establishment, cultural norms, law built up over the centuries, etc. You say entering the unlocked bar wouldn’t be a trespass, but that isn’t even strictly true. If you entered during open hours, no trespass, closed hours, trespass. You would state your case to the judge ("the lights were on, it was 12 o’clock yadda yadda) and the judge would decide if there was a trespass committed.