So many angles to address:
Your own fecal matter is not dangerous on average; nor is the fecal matter of most other individuals.
The pathogenicity of orally-ingested organisms typically depends on the nature of the organism itself, along with the quantity. It doesn’t depend on whether the source was stool or contaminated food or drink.
Most ingested bacterial gut pathogens produce illness not related to bacteremias (getting into the bloodstream), but to effects of the the pathogens multiplying in the gut itself and secondary effects of toxins produced by the pathogens. Bacteria which get out into the bloodstream from the gut typically get there by a defect somewhere in the intestinal lumen (especially in the large intestine and especially near the exit point); and also from the oral cavity which has natural entrance points because of the teeth, exacerbated by mechanical events such as chewing and brushing, and local perturbations in the mouth such as gum disease or abscesses.
As a rule of thumb, typical gut-entry bacteremias (we pretty much only call it a bacteremia if we can culture the offender from your blood) are organisms which are benign in the gut but bad in the bloodstream. Among other problems, they can find a secret place to live (say, an abnormal heart valve or some other place where the endothelium is roughened) and once they gain a foothold inside the body, cause problems. The most common of these organisms are your own, to which you are exposed all the time; at a microscopic level we all get bacteria in our bloodstream with our own organisms every day when we brush our teeth or poop. It doesn’t make any sense to call your stool dangerous. What’s dangerous is a crappy lumen, or an anus abused by hard stools or hard…whatever.
Among the most stupid things we do as a culture (no pun intended) and as clinicians is get too hung up on coliforms. They are everywhere, and what prevents them from giving us trouble is not bleach but host defense. While it is possible that host defense could get overwhelmed by eating a whole plateful of crap, it’s an exaggeration to say fecal matter per se is “dangerous.”
What about pathogenic bacteria, then? Well; they are dangerous (and particularly so to a naive host), which is why they are called pathogens. And those pathogens are the real reason to have modern sanitation and hand washing–both of which tend to prevent unrestricted spread of fecal-oral illnesses.
Interestingly, it turns out slightly filthy people probably have more robust immune systems than ocd-level bleachers. So keep your hands washed and fist-bump for the most part, but periodically reach out and touch the homeless guy under the bridge.