Modulation is changing one signal or medium to carry the pattern of another.
For example, an AM radio signal with a frequency of 1000 kHz can be controlled so that its broadcast power goes up and down slightly in time with an audio signal that ranges from 100 Hz to 15 Khz. They call changing the broadcast power modulating the amplitude, or Amplitude Modulation, or AM.
In FM Radio, it’s similar, except instead of changing the power of the signal so that it pulses in time with the audio signal, the frequency is varied (or modulated) by a small amount.
Similar things happen with television signals. And when doing analog audio recording, magnetic mediums (tape or disk) have maximum amounts of signal they can store.
In all these cases, the modulation or change to the signal is supposed to be restricted to a certain range, both for purely physical and practical reasons. You can’t change an AM radio signal’s power so much that it goes below zero, that’s impossible. You don’t want to vary the frequency of an FM transmission so much it interferes with a station broadcasting on the next base frequency over.
So the receiving or playback equipment (radios, TVs, players, etc.) is built to decode signals that have been modulated within these expected ranges. If a signal is “over modulated” it has been changed too much from the baseline. This can have various effects depending on the severity of the error, and the receiving or playback equipment.
Some of the effects are similar to clipping, but some can be very different. An over modulated TV signal for example, will often cause a varying “buzz” in the audio, because the video portion of the signal is being pushed into the audio portion.
On edit: I see fishbicycle has given a much better explanation of the term in how it is normally used in context of analog tape recording. I’ll leave my two bits in just for general explanation.