Oddly, we find our best spice prices in the bulk section of Whole Foods market. This may be partially because the only “little ethnic store” that was around our old house actually did not sell anything terribly fresh and/or amazing. Just larger bins of spices.
But the Whole Foods bulk section is great because you can measure out as much or as little as you need, and the per ounce price is a great deal better than your average grocery store containers (this includes their containers, as you can imagine). We got enough bay leaves for several soups for pennies, because it was so light. We got some paprika for not much more money, same reasons. Best of all? It’s not likely to go bad, because we purchased so little. If, for some reason, we don’t use paprika as often as we expect and end up tossing it, we’re tossing very little spice or money.
I have not tried any online stores, admittedly, so I can’t speak to that.
In the category of herbs & spices you may wish you had on hand, I don’t have a number of suggestions, but I’ve got a few. I love having a few bay leaves on hand, and paprika, as I mentioned above. Some curry powder can be fun, even for dishes that aren’t typically curries. Cinnamon, cloves, alspice, and nutmeg get constant rotation in my kitchen, though I possibly bake more than you do. Mace is occasionally nice to have around, but I would DEFINITELY get it from some sort of bulk container, as it’s not the most common spice in dishes.
My partner & I also use a lot of powdered New Mexican chiles, but that might be harder to get your hands on. If you want some, feel free to PM me, and I’d be happy to arrange for you to get some.
ETA: Oh, and it may be worth it to you to go to a place like Container Store or something and get an empty spice rack if you’re buying from online sellers or bulk sections. Old mason jars & other canning jars also work fairly nicely for storing things like this.