24 is a pretty small capability. They are overdue to replace the former Soviet aircraft still in their inventory. Leaning forward in the saddle to upgrade to a fifth gen fighter instead of just buying more fourth gen F16s as replacements is probably a smaller issue.
Air defense improvement like the ongoing build of the Aegis Ashore in Poland along with the purchase of the Patriot are a much bigger punch in Russian teeth. Shifting a good chunk of their armored force to the German built Leopard 2 and the upgrades of the 2A4 variants they have to the new 2PL variant are probably bigger kicks in Russian teeth too. NATO allies can flex modern aircraft forward a lot faster than they can ship heavy ground forces by sea
Still it’s a nice little addition for them and their ongoing modernization effort. Poland is quietly becoming the cornerstone of NATO’s deterrence in eastern Europe.
Serving as stealthy, forward air defense sensor nodes is going to be incredibly valuable, and synergizes well with what you mention in the beginning of your second paragraph. They don’t need to fire a shot (though I imagine they’d be good at it), provided Poland gets all of the networking toys with the platform, and thereby can offload target information to the shooters.