The expected value thing doesn’t quite work here, as you’ve noted. Rather, suppose you kill N enemies, each of which has an 85% probability of dropping 4 items and a 15% chance of dropping 8 items. Then the cumulative probability of getting no items at all when killing N enemies will be
(probability of exactly 4M items being dropped)*(probability that 4M items contain no rare items)
summed over all M from N to 2N. (Note that the number of items dropped is always divisible by 4.) The probability of exactly 4M items being dropped after killing N enemies is (if I’ve done my math right)
(0.15)[sup]M - N[/sup](0.85)[sup]2N - M[/sup]( N choose (M - N) )
and the probability of 4M items containing no rare items (assuming all “drops” are independent) is (0.991)[sup]4M[/sup]. (The number 0.991 is the probability of a crap item being dropped, given the probabilities of the good stuff.) So you need to calculate the number
(0.991)[sup]4M[/sup](0.15)[sup]M - N[/sup](0.85)[sup]2N - M[/sup]*( N choose (M - N) )
summed over M from N to 2N. This is equivalent to
(0.991)[sup]4N[/sup] (0.85 - (0.991)[sup]4[/sup]*0.15)[sup]N[/sup]
without any summation necessary. (Expand out that second factor as a polynomial to see this.) You can then subtract this number from 1 to find the probability of getting at least one rare item. This gives a 64.6% chance of getting a rare item after killing 25 enemies, an 87.4% chance after 50 enemies, and a 98.4% chance after 100 enemies.
Whether or not this answer changes under a “no multiple rare drops” rule depends on how said rule is implemented. As long as the probability of 4 given items containing no rare items is still (0.991)[sup]4[/sup], then the above calculation still goes through. It’s easy to envision a software rule that would give a result like that — something like “if there are multiple rare items, replace all but one with crap”. The only wrinkle I can think of is when the enemy drops 8 items; are these regarded as a single set of 8 by the software, or as two sets of 4? If the latter, then the above result goes through; if the former, then the probabilities of getting rare items will be diminished somewhat (though, as iwakura43 points out, the effect would be very small.)