The water is heavy, so it should flow out… but as it starts moving down, it stretches out and the water pressure at the top gets less. (Or if there are any tiny bubbles up at the top near your finger, they expand as the water column moves down, and again the pressure there gets less.) So the pressure between the top and bottom of the water is not equal as the OP thought.
That lowered pressure is too strong. The pressure difference between the outside air and the top of the water column is more than enough to keep the column lifted up against your finger.
So, make the straw longer. That will make the water column heavier. How tall must it be before the water column finally starts falling? That would depend on the outside air pressure, so results will be different on a mountaintop, versus at sea level, versus down in a pressurized submersible.
At 15PSI pressure, the water column ideally starts falling when the column is 33 feet tall.
But if you actually try performing this experiment (with 50ft of aquarium tubing in a stairwell!) you’ll find that you don’t need 33 ft. This is because water is full of dissolved air, and if the pressure starts falling, the water “effervesces.” The top of your aquarium tubing will fill with bubbles and the water column will start descending. If you want to see the “33 feet effect,” you have to use degassed water.
The classic water-column experiment doesn’t use water, it uses a much heavier liquid: mercury. If you dunk a glass tube into a container of mercury, seal the top of the tube, then lift it upwards, the mercury column will stay up there unless the column is taller than about 29 inches. Ever hear the phrase “twentynine inches of mercury?” Probably, since you can use a mercury-filled plugged-straw setup to measure the pressure of outside air.
PS
No matter how hard you suck, you can’t create more than 15PSI of “suction” at sea level. Put a vacuum hose on your neck and it will produce a mild hickey. But if you take yourself deep underwater where the pressure is higher, the hickey-producing ability of vacuum pumps becomes proportionally greater. At the bottom of an ocean trench, a vacuum pump could suck flesh into a hose like it was whipcream, or produce a hickey in a steel plate. I’ve always wondered if that’s why giant squids are able to leave scars on whales. At 1 ATM pressure I wouldn’t think a 1" squid sucker could make much of a mark. But 33ft underwater the pressure is two ATM, and 66ft deep it’s three.