I addressed your objection in the OP. Sure there are shared moral principles, but this does not mean that (i) everyone applies them in the exact same way (a weak argument, I admit, as your poison ivy example points out) and (ii) everyone accords the same punishment to someone who acts not in accord with their moral principles.
Also, people debate about whether an action is immoral even when it would be impossible or unlikely for any (or many) other people to find out that the person did the action. Therefore, violation of a shared moral principle with assumed consequences does not support all arguments about whether an action is moral or not.
Also, people debate about whether an action is immoral even when lots of people do that action without apparent consequences (or with mildly bad consequences). For example, the many threads on whether it is immoral to drive an SUV.