Goyim as slaves.

I can certainly imagine that (due to being prohibited from owning land or doing certain other things) Jewish people became stereotypically prominent in businesses where you could make good money without owning any land or “crossing a line” in other ways. Not a majority necessarily, but enough to form a stereotype in people’s minds.

In addition, certain doctrinaire Christians apparently refused to enter the loan business, so there was an opportunity.