@Mangetout utterly nailed it in the first post and more succinctly here in the second.
Those prior threads he cites go into great detail. Worth reading. There are others I think.
Vacuum is only slightly lighter than hydrogen. But hydrogen at ambient pressure of your outside atmosphere at whatever altitude you want to cruise is incredibly powerful at supporting your sphere from the inside. Conversely vacuum provides zero support.
You can make your hydrogen spheres out of the thinnest lightest imaginable plastic sheet, if they’re filled with hydrogen (or any gas really) to the same pressure as outside. But if it’s vacuum inside and atmosphere outside, you need very stiff, relatively thick walls to keep the sphere from being crushed. The extra lightness of the vacuum doesn’t come close to repaying the extra heaviness of the container.
It’s a losing game all the way around.
More cites: