Yes, naita is correct. The acid in the coffee is causing the curdling, and this is more likely to happen as the milk gets closer to being spoiled.
…This is because coffee and tea contain just enough acidity to tip the pH of milk to the point of curdling. The effect is most often seen in milk that is close to going sour or when adding milk to very hot coffee or tea, since the high temperature can coagulate casein.
Side note: you’re making a café au lait. That sounds much more sophisticated than a “poor man’s latte.”