Over here it is now late at night on a Friday. Just home from a pleasant evening.
OK, some guidance. The simple circuit you have doesn’t really convey a lot of what is going on. It is sort of a mix of layout and schematic. The datsheet for the LM380 is available here. (PDF) There are a few points to be made. With a 12 volt supply and an 8 Ohm speaker the maximum device dissipation is a shade over one Watt. This is low enough that you can actually get away with no heatsinking of the package - although good practice and probably improved longevity of the part might suggest that some heatsinking is a good idea. The datasheet suggests that soldering the package to a printed circuit board provides good enough heatsinking if it has enough copper foil. Pins 3,4,5,10,11,12 are all a common ground pin - and soldering all these to a PCB where there is a nice slab of copper foil connected to them will provide significant extra heatsinking, enabling higher power dissipation… With your application I would not worry greatly. (I have seen applications where copper foil has been soldered directly to pins 3,4,5, 10,11,12 - but that was to make an amp using a much higher power supply voltage.)
Where you need to be careful is in the grounding and layout. The datasheet is clear that the power supply runs must be not be more than a couple of inches before you need the 0.1uF decoupling capacitor on the power pin. There is a clue here that the capacitor should connect right next to the pin on the IC, with minimal lead length. If you were constructing this dead bug style (ie the IC on its back and directly soldering to the pins) you would solder right to pin 14. The other end of the capacitor should go straight to ground.
Ground is the critical issue. The layout you have does not convey the crucial point. There needs to be one point in the layout that is the designated ground point. Ground is not a diffuse idea that you can link together all over the place. The place of this single point is pin 7, or a point right next to it. Everything in the schematic that is shown running to ground should connect directly to this point, and not go via some other circuit element that also is shown connected to ground. So pin 3 is connected directly to pin 7. The decoupling capacitor goes straight to the ground point, the shield of the input wire, the bottom leg of the volume potentiometer, the -ve connection to the speaker, and most importantly, the 0 volts of the power supply. Clearly this is going to get crowded, which is why it doesn’t actually all connect to pin 7, but all these runs should connect to a point nearby. Such layout is commonly called a star ground. It eliminates a host of problems, and I suspect that it will eliminate any stability problems your amp has, and remove the distortion issue. There is a good diagram at the bottom of this page, (although they show pin 6 coupled to ground, I would not do that. They have also missed the needed connection of pin of of 3,4,5,10,11,12 to pin 7/Ground, but the ideas are there.)