One thing to be careful of when buying books “by HP Lovecraft” is to make sure that the book is actually by HP Lovecraft. Lovecraft wasn’t the sole originator of the Cthulhu Mythos, it was a shared world created by a number of authors such as August Derleth, Robert Howard, and Ramsey Campbell. Lovecraft himself was an enthusiastic collaborator, who worked with dozens of writers and would be writers (including Harry Houdini) in his career. However, there is no doubt that Lovecraft was the ringleader and primary creative force behind the Mythos. The downside to this are all the story collections billing themselves as “The Horrors of HP Lovecraft,” which contain only one of his more well-known stories and a bunch of Mythos stories by people you’ve never heard of. Not that this is always bad; it’s a good way to get introduced to new authors, but it is annoying when you’re specifically looking for stuff by Lovecraft. More annoying are the less-than-scrupulous publishers who have printed new editions of works by lesser writers, such as Derleth, under the name HP Lovecraft, because the book mentions Cthulhu once or twice. The actual author often won’t be listed on the front cover at all.
In the interests of helping you avoid the hangers-on, at least to begin with, here are the major short stories that every Lovecraft fan is required to have read, lest the Shoggoths come and take away your membership card:
“The Call of Cthulhu”
“The Haunter in the Dark”
“The Dunwich Horror”
“The Whisperer in the Darkness”
“Shadow over Innsmouth”
“Pickman’s Model”
“The Colour Out of Space” (My personal favorite of his stories)
“The Rats in the Walls”
“Cool Air”
“The Terrible Old Man”
“At the Mountains of Madness” (Also my personal favorite of his stories)
“The Shadow Out of Time”
“The Case of Charles Dexter Ward”
“The Dreams in the Witch House”
“Randolph Carter’s Statement”
These are available in literally hundreds of short story collections, but all can be found in the books The Dunwich Horror and Others and At the Mountains of Madness, both of which, AFAIK, are still in print.
Lovecraft also wrote some interesting things about horror literature. Check out Supernatural Horror in Literature if you’re at all interested in that sort of thing.