Our models of Big Bang nucleosynthesis are quite strong, and hence our confidence in the amount of baryonic matter is quite high. No dark matter model is anywhere near strong enough to significantly shake that confidence.
Black holes might be some or all of the dark matter, but then you have to ask where they came from. The baryonic-matter limits still apply, so either the holes would have to have been formed from something other than baryonic matter, or they would have to have been formed before the era of nucleosynthesis, very early in the Universe’s history. And if primordial, low-mass black holes are really so abundant, then you also have to ask why we’ve never detected any of them. This is a problem in general with the more exotic hypotheses for dark matter: The more exotic something is, the more likely it should be that we would have detected it in some other way.
And the microlensing MACHO searches are good at least as far down as planetary masses (and would also, incidentally, detect black holes in that mass range). I don’t know how large dark-matter objects could be without producing the fog effect.
Put it all together, and we don’t know that the bulk of the dark matter is made up of stuff that doesn’t interact electromagnetically at all, but it seems by far the safest way to bet. It’s really not so surprising, after all, that there would be some such particles, and if they exist, there would be a good reason why we don’t easily detect them.