From a first principles point of view the shuttle isn’t a trivial problem. As pointed out above, the thing moves, and fast. Also, even when it is directly above, it is still 300 miles up. That is a long way for a radio signal to go. Especially given the small size of the antennas available, and since it is moving, the low gains possible.
Your cell phone signal travels a few miles. Ideally the signal drops off with the square of the distance. So in comparison the Shuttle’s actual signal strength available isn’t going to be exactly huge.
The limiting factor for the quality of communication is the information rate. This is the product of the bandwidth of the signal and the signal to noise. We have seen how the signal strength can drop. The noise is essentially impossible to reduce, and we are left with the bandwidth of the signal. If you use a simple analog radio - i.e. AM or SSB, or a standard narrow band FM, the bandwidth is fixed. And that is about it. Higher quality communications must increase the information rate. And that means at least one of: wider bandwidth, more signal, less noise. Digital communications are identical. It is just another form of modulation. Your GSM mobile phone has exactly the same issues. It is just that digital has rather more brutal failure modes. Breakup of the sound, garbling, total dropout. Whilst analog just gets noisier and harder to understand. But humans being what they are are better at hauling meaning out of a very noisy analog signal than a juddering, breaking up, stuttering digital signal. Even if the overall information content is the same.
When you want ot run a safety critical system, you want the simplest and best proven technology for your comms. There is no gain to be had going to a more complex system, and considerable risk.