Let me attempt to add something helpful:
With the possible exception of “free” jazz, there is a form that is being followed. Jazz is “advanced” music (nowadays, anyway) and sometimes the form is difficult to discern, but it is there.
When I first got interested in jazz, I was already a trained classical musician but even so, I could not hear the form at first. I believed, incorrectly, that they really were just making it up as they went along.
Not so. The improvisation is based on the rhythmic and chord structure of the “head” - the tune itself, usually played through once at the beginning and at the end. In between are the improvised choruses, which depart from the melody but retain the original chord structure. Each improvised chorus is exactly as long as the “head”, and the chord changes take place at the same points.
This may be nearly impossible for a beginner to hear, because chords are improvised on so freely, using harmonic extensions to the basic chord. Example: in classical music, a C major chord is just C-E-G. Jazz adds the seventh (B) or the sixth (A) for starters, then proceeds upwards from there, adding the ninth (D), the eleventh (F, usually modified to F# in a major chord) and the thirteenth (A again). Taken all together, these extensions add up to an entire jazz scale based on C major, which is what’s improvised on when the chord is a C. The beginner hears these extra notes as “outside”, but through repeated use they become “inside”, which is why postmodern jazz improvisers are always seeking to go further and further “outside”, using notes that sound clashy even to an experienced jazz listener. Personally, they lose me here: my harmonic tastes are solidified circa 1961.
The “sharp eleventh” - F# in a C chord - used to be a very “outside” tone that outraged the traditionalists in the 50s. Now, it’s practically banal.
Anyway, my main point is that they’re NOT “making it up as they go along.” (except in free jazz.) They know exactly where they are in the form - at any given point, the musicians could sing the melody that would be taking place over a particular chord - they “know where they are in the tune.”
As for rhythm - yes, it can be stretched and “random” sounding at points, but the underlying beat never changes. Everybody knows exactly where each downbeat is, even if it’s just implied and never actually played. A great example of this is the four classic Bill Evans trio records with Scott LaFaro - they hardly ever actually play on the downbeats, but they are in perfect agreement on where the downbeats are. They dance around them, because playing right on the beat is too “obvious.” Sorry if this sounds pretentious.
The form of the entire tune is preserved during improvisation. If the melody was 32 bars long, each improvised chorus will be 32 bars long - you could time it with a stopwatch, if you’re unable to actually count along.
Try listening to a jazz recording of a tune you know. Sing the tune multiple times while listening to the improv. You’ll find that it matches up; the improvisation goes with the melody, round and round and round. Of course this is not done too obviously; jazz players are always pushing the envelope.
Lisa Simpson: “You have to listen to the notes they AREN’T playing.”
Some Guy: “I could do that at home!”