It seems to have started in monasteries, when a distinction was observed between those who had obtained priestly ordination, addressed as “father”, and those who hadn’t, addressed as “brother”. There were differences in both function and status between them, hence the reason for having different forms of address/reference, and the choice of terms drawn from family relationships reflected the view of the monastic community as a spiritual family.
Over time, usage seems to have crossed over to refer to/address priests who weren’t monks. Note that this wasn’t a uniform practice; in Italy, which is about as Catholic a culture as you can get, for example, a priest was (and I think still is) addressed/referred to as Don X, where “Don” is an honorific with no connotations of paternity (and one which is not exclusive to the clergy).
In French, a priest is addressed not as père, the conventional French word for “father” but as abbé, which comes from a Syriac root which in fact does mean “father”, though most French-speakers probably wouldn’t immediately make the connection. Again, the title came through the monastic tradition; compare the English “abbot”, a spiritual father to his monks.
In German I gather the distinction is still maintained; a monastic priest is addressed with the latin term pater, meaning “father” while a non-monastic priest is pfarrer (“shepherd” or “pastor”). The German Vater (“father”) is not used.
The Spanish term padre (“father”) is sometimes used by English-speakers, particularly in military circles, to refer to a priest. I don’t know whether it is so used by Spanish speakers, though.
In short, the title was originally monastic, and reflected monastic self-understanding as a spiritual family. Over time it came to be widely, though not universally, applied to non-monastic clergy as well. I suspect there was a time when monastic clergy had a higher status, and non-monastic clergy were seeking the same status, but that’s just a guess on my part.