I happen to be near mine, and we’re both wrong: Gandalf doesn’t agree with Elrond’s assessment. Elrond merely says of Tom that “maybe I should have summoned him to our council” – and Gandalf says the guy would not have come. Erestor then picks up the conversation by suggesting they send messages to Tom and obtain his help, since Tom seems to have a power even over the Ring; Gandalf then disagrees, preferring to say that the Ring has no power over him; Erestor then asks about giving the Ring to Tom, and Gandalf then makes the claim about Tom being too unsafe a guardian for the aforementioned reasons.
And then Glorfindel and Galdor take up the discussion for a bit (including the conclusion that power to defy their Enemy is not in him, unless such power is in the earth itself) – but Elrond never actually weighs in on Tom’s alleged impracticality or detachment or whatever.
So IMHO it’s just Gandalf’s read of Tom. And IMHO what Gandalf doesn’t know about Tom gets helpfully summed up long after the Council of Elrond (“I am going to have a long talk with Bombadil: such a talk as I have not had in all my time. He is a moss-gatherer, and I have been a stone doomed to rolling. But my rolling days are ending, and now we shall have much to say to each other”), as does what Gandalf hadn’t known about – oh, say, what’s in the earth (“it has a bottom, beyond light and knowledge … We fought far under the living earth, where time is not counted. Ever he clutched me, and ever I hewed him, till at last he fled into dark tunnels. They were not made by Durin’s folk, Gimli son of Gloin. Far, far below the deepest delvings of the Dwarves, the world is gnawed by nameless things. Even Sauron knows them not. They are older than he. Now I have walked there, but I will bring no report to darken the light of day”).