The correct answer has in principle been given – the atomic nucleus tries to reach a lower energetic configuration if possible, or, to avoid attribution of motives to inanimate particles, some configurations of the neutrons and protons within the atomic nucleus have the intrinsic property that a slight change causes the particles to fall into a lower energy state, releasing energy in the process.
Think of it as an unstable equilibrium – small changes, brought on by quantum fluctuations, cause the down slide into a more stable potential valley, much like a pencil balancing on its tip is essentially in a state of equilibrium – gravity pulls in all directions equivalently – but a small disturbance (arbitrarily small, in principle) will cause it to shift and fall over.
Let’s look at the stability/instability conditions for the three most commonly known decay modes, alpha, beta and gamma:
Alpha decay: A nucleus emits an alpha particle, consisting of two neutrons and two protons (a helium nucleus, in other words). Now, why would any self-respecting atomic nucleus do such a thing? Easy: because the combined energy (mass-, binding- and kinetic energy) of the alpha particle and daughter nucleus is less than that of the original nucleus. (It should be noted that, of course, the total energy never changes; it merely gets ‘more evenly distributed’, hence, entropy increases.)
However, this is obviously not true for all nuclei, otherwise they’d all alpha-decay until there’s nothing but helium left in the universe. What we’ll have to look at here is the binding energy per nucleon: an atomic nucleus is an energetically more favourable configuration of nucleons than just the isolated neutrons and protons, meaning, essentially, the mass of the nucleus is smaller than the sum of the masses of the individual nucleons. That difference is called the binding energy of the nucleus, and is provided by the strong nuclear force, which binds the nucleons together, offset by the Coulomb (electromagnetic) repulsion between the protons (like charges repel). To reiterate, you have, in the nucleus, two opposing forces, an attractive force that acts between all nucleons, the strong nuclear force, and the repulsive Coulomb force that acts only between protons. This nets an attraction for all stable nuclei. However, both forces have a different range characteristic: the strong nuclear force falls off much quicker than the Coulomb force, meaning it has a shorter range. Now, as nuclei grow progressively larger, the contribution of Coulomb repulsion eventually grows larger than that of the strong attraction, eventually overcoming the latter and making stable nuclei impossible.
As a further consequence of the different range characteristic between the two opposing forces, there exists a maximum binding energy per nucleon: starting at small nuclei, the binding energy per nucleon increases in the direction of bigger nuclei, owing to the fact that each additional nucleon is attracted by all the other nucleons already present (this is within the short reach of the strong force). Eventually, though, the nucleus grows large enough for the Coulomb repulsion to become dominant, and thus, the binding energy per nucleon decreases with each additional one, leading to a curve like this one, with nickel/iron at the maximum. Thus, nuclear interactions (fission/fusion processes in this case) always occur in the direction of increased binding energy per nucleon: light elements can be fused, i.e. smashing two light nuclei together will result in a heavier nucleus with a larger binding energy – and thus, lower mass and an energetically favourable configuration – than each of the original nuclei, while heavier nuclei are generally fissile, i.e. splitting a heavy nucleus into two lighter parts (which is essentially what occurs with alpha decay) yields a favourable configuration, since both daughter nuclei will have a larger binding energy per nucleon than the original nucleus, and thus the sum op their masses is smaller.
Alpha decay has a further complication in the form of the so-called Coulomb wall: essentially, an alpha particle is trapped within a potential well inside the nucleus, described by the combination of attractive strong/repulsive Coulomb force. Classically, it can’t get out of there, but quantum mechanics allows for a non-zero probability of just ‘appearing’ outside that well (‘tunneling’), which accounts for the statistical nature of the decay (at any given point in time, there’s a certain probability of it happening, but it’s impossible to predict exactly when).
Beta decay: There’s two kinds of that, beta-minus and beta-plus decay. We’ll only consider the first one, the other works much the same way. Beta- decay is the conversion of a neutron into a proton, while emitting an electron and an electron-antineutrino (conversely, beta+ is the conversion of a proton into a neutron while emitting a positron and an electron-neutrino). To better understand this, we’ll have to distinguish between the two nucleons in the nucleus, neutrons and protons. Neutrons are subject to the strong interaction only, while protons both experience strong interaction and Coulomb repulsion. Hence, you need generally more neutrons than protons in a nucleus to glue the whole thing together (i.e. aim for a maximum of binding energy per nucleon), at least for sufficiently large nuclei. Thus, in some cases, it’s energetically more convenient to transform one nucleon into another, and so we get something that’s called the ‘valley of beta-stability’, within which lie those nuclides that have the optimum pairing of neutrons and protons, and to its sides are those with either too many neutrons or protons, which are likely to undergo some form of beta decay.
Gamma decay: Well, that’s essentially a bit of a different beast than the other two, as there’s no transmutation happening (i.e. a nuclide that has undergone gamma decay still is the same nuclide, just with less energy). For a simplified picture, just think of a nucleus that has, for some reason, an excess of energy (most likely after undergoing alpha- or beta decay), that it wants to get rid off, which it does by emitting electromagnetic radiation (photons, here commonly called gamma quanta). It’s a bit more complicated than that, but I think I’m just about done writing now.
I hope I didn’t fudge the details too much, if I did, I’d welcome any corrections.