You can drop anything that goes in the pill/tarot card slot, and trinkets (except for The Tick, that can never be dropped) by holding tab or right trigger. But any non-usable items you pick up are yours for life. That’s why I generally avoid Bob’s Brain (the projectile bomb you mentioned). On a similar note, if you ever get the shop items that let you hold 2 pills or tarot cards at once, tapping tab or right trigger switches between them.
I had an Azazel hard mode run to The Dark Room last night that was going quite well. Until I picked up Dr. Fetus (that turns your tears into bombs, and really sucks) and Orbit, or whatever it is that causes your tears to shoot in a circle around you. I picked them up pretty near to reaching The Lamb. I was quite fucked after that. But, i’m glad I took the risk and tried those items. Dr. Fetus truly is shit though.