The Russians aren’t gonna build any more high-tech aircraft, the Europeans could but don’t want to pay for them, the Chinese might want them but are still too poor and too far behind technologically.
And what exactly is the mission of a stealth plane? To avoid detection by enemy radar so you can go in and bomb your target without getting killed by anti-air defenses. Before a country goes out and builds a stealth plane, they have to ask themselves how likely it is that they’re going to be bombing a heavily defended enemy country vs the cost of building that stealth plane.
The US developed stealth planes in anticipation of bombing the Soviet Union, which had the toughest air defenses in the world. The Soviet Union is gone, but we still have the planes, the DoD isn’t going to throw their new toys in the trash just because their intended target is gone. Other countries either can’t afford them or don’t want to pay for them. Who are you going to bomb with them?
Remember that stealth is a continuum. Against a third world air defense system our stealth planes are very effective. Who knows how they would have done against the Soviet air defense system? And Chinese stealth planes probably would fare very poorly against the US air defense system, although some stealth is better than no stealth.
If the plane was approaching at just under the speed of sound, the effect you describe would happen whether it was stealthy or not, i.e. you wouldn’t hear it until a few seconds before it passed.
If it was flying faster than sound, you wouldn’t hear it at all until after it passed.