Maybe you don’t have a problem with linking to HTML pages, but it’s the same thing, as far as the law and the software are concerned. The one difference is that images are generally loaded automatically by the browser (though some browsers don’t load images, and most can be told not to), and links are generally only loaded by user interaction (though spiders and some caches will automatically load all links).
As far as having a problem with it, someone could certainly claim that linking to a sub-page, instead of the front page, robs them of banner ad revenue.
Or they could go with the bandwidth leech angle - B makes a link from his site to A’s site, which causes Googlebot and other spiders to automatically load A’s site after it encounters B’s site. The link on B’s site causes spiders to use A’s bandwidth, just as an embedded image on a leech’s site causes browsers to use your bandwidth.
How many people check the URL of a link to see if it will take them to a different site, or just a different page on the same site?
The question is whether they need permission at all. Do I need permission from the Chicago Reader to post a direct link to this thread, instead of saying “Go to the SDMB, click Great Debates, then click ‘Hatch says destroy the downloaders computers’?” (Whoops, I did it anyway.)
In both cases, the URL is simply a way to get the user’s computer to save him time. The author of the site that’s using your images could have said, “Go to yosemitebabe’s web site, click the third link on the left, and look at that picture. Now imagine it being right here.” Instead, they gave the user’s web browser a shortcut so the user wouldn’t have to go out of his way to do it.
There is, of course, an additional question of whether the other site’s author is misleading her readers to think she created your picture, or holds the copyright on it. If she takes credit for your work, or presents it in such a way that the user has no way to know it’s not her work, that’s plagiarism and it’s wrong. But I can’t convince myself to feel the same way about simply linking to the file.
By putting a file on your web server in a public place, you implicitly give the world permission to download it.
At this point, whether it’s legal depends on which state you’re in. (Thanks for the heads-up, TWDuke. I hope vssp.com gave you permission to link directly to that page. ;))
I agree that those other webmasters are jerks. But technically speaking, this is more or less what the web was designed to do, and I’d much rather see leeches fought with technical and social measures, instead of legal measures that may end up crippling the web.

Actually, an <A> link to the image file is more acceptable than an <IMG> link, because the viewer must click on the link first before any bandwidth is stolen. With IMG tags, the image is loaded automatically when the user visits the “third party” site. At least with <A> links the user actually has to click before the image is loaded (which can make all the difference).