Typical jazz textbook on improvisation spends an inordinate amount of time on the circle of fifths and the relationships of chords. Most of this can be turned into a concise mathematical formulation. Musicians often seem to be expected to learn this stuff by rote. Or more often they learn a subset that they like and that subset informs their style and sound. They are also expected to be able to recognise a lot of this by ear.
But improvisation won’t work if you are thinking about the rules. You don’t have the brain space to play in the moment and also worry about harmonising a chord or what to play next. Which is why playing to known chord changes is so useful. Everyone knows where they are going, with enough room to explore.
In classical theory there are a lot of rules of harmony that stem from much the same ideas, albeit from a more constrained pallet.
Just how much you distinguish rules from mathematics is a bit open. Musicians wouldn’t recognise, say, a group, within the rules; a mathematician would instantly.
If you look at one of the seminal textbooks on orchestral arrangement, Principles of Orchestration by Rimsky-Korsakov, he spends the first chapter on Helmoltz’s work. Understanding the nature of sound, harmony and timbre from a scientific rather than rote learnt basis. This strikes to the core of the question of noise versus music.
Not that noise isn’t valid thing in music. Tim Hecker, Ben Frost, Lustmord, and a legion more. All the way up to soundscapes of a wall of harsh noise. Done right and you still have a story arc in sound and emotional energy.
So I remain unconvinced that music is captured in simple mathematics in a useful manner.
Something I think music does have some common ground with is humour. The structure of a joke or other humorous story has an arc that isn’t wildly different to the structure of a musical passage.
Where one sees a tiny relationship to mathematics is catastrophe theory, in so far as tension and release, setup and punchline might be captured as a mathematical catastrophe. Again it isn’t clear this is useful, but as an aid to making sense of what happens it is at least interesting. One needs a metric for tension, something that is likely peculiarly difficult. Some things are simple.
The rule of three in a musical phrase pattern. Phrase, recapitulation, variation. The building block of much improvisation, and the base structure of the simple joke.
An Englishman, a Scotsman and a viola player walk into a bar. The Englishman says, “a gin and tonic please”, the Scotsman says, “a scotch with water on the side”, and the viola player looks around and says, “I’m terribly sorry but I appear to be in the wrong joke”.