It’s not always pejorative, and depends on the culture. Sometimes “baba” can be a child’s affectionate term for a grandmother. But even in cultures where it has non-pejorative connotations, it can be made pejorative by putting an adjective like “old” in front of it. While this is technically redundant, linguistically it emphasizes the “old” part, so ends up with a similar connotation to the English “old hag” or “old bat”.