There are a variety of ways for evaluating the performance of explosives, including measuring their detonation velocity, the amount of work they do on a lead cylinder, and the amount of gas they evolve. Then there’s taking those figures and comparing them to the density of the explosive material. Relative effectiveness (or Relative Strength) is yet another metric. I doubt most chemists would be making many of these compounds outside of an industrial setting.
But for a given value of “homemade”, my guess is that RDXwould be the leading candidate, with a detonation velocity of 8750 m/s and a density of 1.76 g/cc. Although, perhaps PETNis possible to make with improvised equipment? I don’t know, and would not disclose any information about it if I did.
Alternately, nitroglycerin, with a detonation velocity of 7700 m/s, density of 1.59 g/cc, and a Trauzl test of 52 cc/g, would be a suitable choice. Ammonium nitrate comes in at 5270 m/s, and 1.30 g/cc. It is however, effective, cheaper, and/or easier to work with than the preceding explosives.