I’ve seen movies where a plane could cause windows to shatter or a building to shake by flying close. So what speed would be needed to cause actual damage by tearing past?
Generally anything over Mach 1 is going to create a sonic boom, which would have those effects.
You don’t even have to fly all that close either, a sonic boom can cause damage on the ground when the plane is at thousands of feet.
A Brazilian fighter jet went supersonic during a ceremony and shattered the windows of the Supreme court building:
During a subsonic low-altitude pass, the downwash from the wings (especially during a high-g pullup) can create violent ground-level winds. Here you can see a Blue Angels low-level “sneak pass” with a high-G pullup that blows beachgoer’s umbrellas away:
Here’s a lower-speed pullup, filmed for the scene in Top Gun 2 where Maverick’s Darkstar aircraft blows the roof off of a small building at the end of the runway (the Darkstar was digitally added afterwards):
The roof doesn’t move in this take, but the “behind the scenes” material included with the Blu Ray disc shows that for the final movie take, they had loosened the roof from the building so that the turbulence could indeed move it around.
Needless to say, the object must also be large enough to cause building damage. A rifle bullet would make a crack sound as it goes by, but not break your home’s windows.
True enough, but if that something is a tornado/hurricane/cyclone it will cause more than just breaking windows whilst travelling much slower than Mach 1.
The nuclear powered Supersonic Low Altitude Missile - Wikipedia was planned to fly at Mach 3 at tree top level and would kill people and destroy wooden-frame houses just by flying overhead.
Yeah, but the OP was asking about planes.
No, I was just using it as an example. Though I can’t really think of anything else that could realistically hit the proper speeds.
Not being restricted to realistic… how about superhero? Has to do with volume of air, yes? A speeding bullet no. Would The Flash running by do it?
Are we excluding photons and neutrinos? 
And also leave everything in its wake horribly contaminated.
Oddly enough, not according the linked article. It wouldn’t leak much actual radioisotopes, and the radiation flux while substantial would be momentary at the speeds SLAM was expected to fly.