He was transferred from another orphanage in Brazil.
Here is the young Tom Riddle with one of his brothers from Brazil-
Clearly his father abandoned him because he knew Tom Riddle Jr was not really his son, a decision that saved him from being murdered by Nazis, though Voldemort did it himself in the end.