And in most cases literally the packaging.
Sugar naturally in fruits, for example, is packaged inside cell walls as well as along with fiber. Both impact the rate and location that the sugar is absorbed into the blood stream. Remove those things and you functionally are closer to sugar in water.
The “milk matrix” is somewhat similar.
For milk, control of postprandial glycemic responses appears to be multifaceted, including a controlled rate of gastric emptying, a rate of glucose and galactose uptake into the bloodstream controlled by enzymatic hydrolysis, as well as stimulated insulin secretion to enhance uptake of blood glucose from the bloodstream. Altogether, this allows milk to deliver comparatively high levels of carbohydrate with limited glycemic responses.
Added sugar generally has none of those packaging and matrix effects.
I’m not hating on almond milk, but better to eat some almonds. It’s not just all calories either way and I know of no reason to conclude that almond milk is overall healthier.
(Personally I like the taste of plain kefir but that’s just me.)