Worms eat into zombie brains too.
Wow. His inner voice must be LOUD! 
I know this is an old post, but I can offer a little more clarity.
First, yes, the judge is most definitely represented as an ass. This is corroborated by the lines:
“The way you made them suffer, your exquisite wife and mother
Fills me with the urge to defecate!”
Second, while it makes sense that it could be “Good Morning, Worm, Your Honor,” addressing two different characters, the only notable characters present at the time are the judge, the prosecutor, and Pink. Therefore, if this were the case, “Worm” would be referring to Pink, and the last lines from Pink’s wife and mother do not make sense in this context.
Pink’s Wife: “Just five minutes, Worm, Your Honor; him, me, alone.”
Pink’s Mother: “Worm, Your Honor, let me take him home.”
Oh, and the worms both represent death and decay and signify madness. Or, more accurately, the worms represent more of a state if disrepair of the body and mind. Like when something is just left to the elements, untended, unloved, and isolated. Left for nature to gradually consume.
Specifically, Pink has isolated himself behind his wall in order to prevent himself from being hurt as well as to prevent himself from hurting the ones he loves. But this has left
him completely isolated, cut off from everyone and everything that gives life meaning. With nothing of value or substance remaining, he turns inward in an attempt to find meaning. With no
external purpose or desire, he has no incentive to reemerge from his shell, not even to eat or take care of himself. He’s lost the will to live and is just waiting to die, hence “Waiting for the worms.” The line “And the worms ate into his brain” tells us that he’s lost his mind. And, since the worms are coming because he’s neglecting his health, the line is also telling us the reason he’s losing his mind.
As he sits in a catatonic state at death’s door, his mind is coping by stringing together his conscious and subconscious thoughts into a waking dream as it attempts to rationalize what’s happening. This comes in the form of The Trial, where he finally scrutinizes his decision to isolate himself and comes to terms with the fact that it’s not just him he’s hurting. In the end, he finally snaps out of it and tears down the wall.
The animated parts of the film and the album art were done by Gerald Scarfe, or at least based on his ideas, if I’m not mistaken.
There’s an animated scene in the film of someone or something, maybe schoolchildren, being forced into a grinder. If I remember correctly, the strands that come out of the grinder wiggle like worms (homogenous, docile, unthinking citizens). Maybe the judge is addressed as worm in reference to that process.
[Moderating]
I’m not sure how this escaped notice the prior two times this thread was active, but it’s mostly about interpretation of a work of art, which is a much better fit for Cafe Society than Factual Questions. I’ll go ahead and move it.
Oh, and welcome, @Gaius315 . Don’t mind the zombie jokes; they’re just because this thread hasn’t been posted to for a long time.