That’s because
[spoiler]1. Walt doesn’t retrieve the Lily of the Valley from Brock’s house. It is in his own backyard – we see it twice before that:
After Walt’s family goes to stay with Hank & Marie, Walt is sitting in the backyard waiting for Gus’ goons to show up. He has his gun sitting on the patio table, and he is absentmindedly spinning it. Twice it stops pointing at him (a la ‘spin-the-bottle’), but the third time it points away from him. It is pointing at the Lily of the Valley, giving Walt the idea for the scheme (though this is not obvious to the viewers at the time.
After Gus is dead, Walt and Jesse meet on the roof of the parking structure where they shake hands and Jesse tells Walt that Brock will be fine, that the poison was Lily of the Valley, and rationalizes that Gus had to die anyway, right? Walt shows empathetic relief that Brock is OK and agrees that Gus had to go. They drive away and the scene switches to the White’s back yard, where the camera zooms in and focuses on the Lily of the Valley (with a convenient label), revealing that Walt was the poisoner all along.
Walt gathers up the plant, along with the bomb-building equipment, when he’s cleaning up before his family returns in episode 5-01.[/spoiler]
2. It is never made clear how Walt poisoned Brock. Does he give him berries directly? Squeeze them into a juice box? When he encounters Brock later, at Jesse’s house, there’s no indication that the kid recognizes him, so it seems he didn’t hand him anything directly. Who was the surrogate? Mysteries that the writers left as such…
Of course, while we’re on the topic,
When Walt has to get the money to bribe Saul’s receptionist he needs to get into his house, which by now is occupied by Gus’ men. He’s already built his pipe bomb, and he won’t be cleaning up the kitchen until the beginning of season 5. So these two guys, who work for Gus, presumably know that Walt has been “cooking” something up at his house, and the bomb angle seems to be obvious – did they inform Gus of this? If Gus knows Walt is running around somewhere with a bomb, maybe that’s why he doesn’t return to his car. But it seems that Tyrone should be specifically looking for a bomb in Tio’s room – if he has a clue what to look for, he’s likely to find it.