Somewhere-or-other, a long time ago, I read some variation of this story too. It was said that the children, deprived of socialization, died very quickly. (ETA: But no doubt, any modern telling of the story must probably be relying in the same account that the Wiki article quotes.)
Without speech from the robots, the children would not only be robbed of speech development but also other acculturation - know knowledge of all the minor technologies that make life possible for humans and all dependent to some extent on speech to promote it.
Children mimic as part of their learning at a early stage, for the OP the children may mimic the robotic sounds or actions related to the object they are trying to convey.
But language seems to be a secondary communication method, the first may be emotional expression, but emotional expression does overlap into voice, such as crying and laughing. Language may have come out of emotional expression, and developed from there, meaning that language is not natural in its raw form, but development of language later on is natural, but also our very thinking of language as the primary method of communication may involve needed to let go of emotional expression is the primary method of communication. In a situation such as the OP however there would not be adults (using language) for the infants (using emotional conveyance) to make such a transition to language using mimicking
Another issue would be that other infants may not respond well to the emotional needs of other infants, and may actually feed off of the happiness of others, such as taking another’s toy. But the robot care takes may or may not be programed to compensate for this and provide above what may be taken, along with discipline and instruction.
The silent-caretakers tale is in Herodotus’ Histories (5th century BCE), and the language-primaeval is Phrygian.
The experiment with human caretakers would be much better than with robots. The children could be given as much love, care, and interaction as any normally brought-up child - but just without language.
My guess is that the kids would develop a language among themselves. This would be much simpler in vocabulary and structure than any existing living language but would allow them to have more or less normal, healthy, relationships.
I found it; it was James IV, not sixth (a forgivable error, if I were dyslexic).