Don’t forget that it’s not just the family. Remember Stewie at Lois’ aunt’s mansion ordering the servants to cut his milk and fight to the death. They sure understood him.
Also, Stewie was in an episode of “Kids say the darnedest things” or the like, and Bill Cosby’s doppleganger understood him.
What about that episode where there was some classroom in the future who just watched an episode of the show. The teacher asked the class if they had any questions and one boy replied something like “Yeah, uh, so do they understand what the baby is saying?”