I came across an interesting discussion on this when listening to a podcast.  Basing Roe on privacy really is not the stretch it has been made out to be. The bits quoted below are from an auto-generated transcript so it reads and are punctuated a little oddly:
S3: Everyone has a reason or a theory of how they could have decided Rose that this protection would actually be stronger. There’s this whole idea. If anyone knows anything about Roe, it’s that it’s grounded in privacy. And privacy isn’t really a great grounding for this, right? This right is too important. And I think that this has been such an interesting thing to watch people say, because, first of all, I think that if you look at Justice Alito’s opinion like this is obviously not about the practice of law. It’s not really about the interpretation of of the Constitution. It’s about going backwards in time. But more importantly than that, I actually think that we have done ourselves a disservice in the way that we’ve talked about privacy. It’s grounded in privacy because the privacy aspect of the 14th Amendment that they’re building on is something that was being built on in that time in the Supreme Court’s jurisprudence. So it was like the most likely place that they were going to continue this. And Sarah Weddington, who argues Roe in front of the Supreme Court, knows this. She offers it amongst her arguments. But I also just wanted to say, is that when you talk about it within the bounds of privacy, within the idea that this is a decision that a woman makes with her doctor, it kind of has this element of being like a little bit shameful and it doesn’t sound like a very strong. Right. But when you think about the 14th Amendment, it’s about family autonomy. It’s about how to construct your family in the way that you want. And it seems impossible to me to argue that you cannot have control over that and you can have liberty and life and happiness under the Constitution. So the thing that I have come to is really feeling like when we say, oh, privacy wasn’t the right thing and oh, I could have decided Roe better and it could have been stronger. I don’t think that there was a legal reasoning way where this was not going to blow up into the fight that it became. But I also think that there’s a way that we can think about this ruling and how it was grounded and actually understand that as a really deep, fundamental right.
< snip >
S4: I mean, he’s obviously someone who has never played Jenga. So there’s that. Yep. Let me just first say, this whole unenumerated rights versus textual rights, that is kind of the spine of this opinion. Again, I mean, conservatives love talking about the fallacy of unenumerated rights until they want to defend executive privilege or qualified immunity, both of which are not explicitly mentioned nor protected in the Constitution. So if we’re going to be consistent about this, let’s be consistent and not itinerant. If you don’t like unenumerated constitutional principles, then don’t like all of them, not just this one. That’s my one hobbyhorse. I have more. The other thing that needs to be said here, the idea that unenumerated rights are somehow our constitutional apostasy is really divorced from the text of the Constitution itself. In his opinion, Justice Alito speaks of text based rights, and he speaks specifically of the first eight amendments as though there aren’t other amendments, including two other amendments in the Bill of Rights Amendment, including the Ninth Amendment, which explicitly says just because we’ve enumerated something doesn’t mean that that exhausts what is available for protection here. Like there might be more. And the fact that we didn’t write it down doesn’t disparage the existence of those other rights. So that suggests that there could be implied rights and indeed there have been implied rights. Justice Alito, in this opinion, says that the 14th Amendment says nothing about abortion. It doesn’t say anything about marriage, does nothing about contraception, parental rights, none of that. My colleague at NYU, Peggy Cooper Davis, wrote a marvelous book many years back, Neglected Stories, The Constitution and Family Values, in which she provides primary source material that makes clear that the drafters of the 14th Amendment were consciously trying to repudiate the institution of slavery and also all of the vestiges of that institution, including the fact that enslaved persons lacked family integrity. Their families could be sold from them, their children could be sold from them. They weren’t allowed to marry. They had no bodily autonomy. They could be sexually compromised by an owner or mated with another slave for the purpose of reproducing the slave population. And so the 14th Amendment’s guarantee of liberty, Peggie argues, was meant to address all of those things, those liberties that were absolutely denied to individuals who were enslaved. And so when Justice Alito says the 14th Amendment was about repudiating slavery, he’s right. He just misses what repudiating slavery actually means. So I don’t buy this unenumerated rights versus text based rights fallacy. And third hobbyhorse for me is this idea that you can sequester the question of abortion because abortion, quote unquote, destroys potential life and that somehow does not imperil these other rights like same sex marriage, interracial marriage, contraception. That is absolutely nonsense. And he’s disingenuous to suggest it and he knows he’s being disingenuous. You see, those same rights are all undergirded by this right to privacy. If you tug on privacy in ROE. You are tugging on all of them. And even Justice Alito has made clear in his own writings that these things are inextricably intertwined in the Hobby Lobby decision from 2014. He argued that certain forms of contraception like IUDs or the abortion pill protocol that just was describing those two are abortifacients because they destroy potential life by preventing it from being implanted in the uterine wall. So it’s how can you distinguish between the abortion that destroys potential life when you’ve already said that these other forms of contraception also do the same thing? How can you distinguish between the unenumerated right of abortion and the unenumerated right to marry a person of the same sex? You can’t. And he knows that. So he’s saying this now. But in that opinion, there is an entire blueprint for eventually coming back to these other rights. And in fact, if you look at it, it says that when legislatures make these decisions, reviewing courts have to accept them because they have to respect the judgment of state legislatures on issues of social significance and moral substance. What could he be talking about other than abortion, contraception, marriage? These are those issues of social significance and moral substance. And he knows that it.
SOURCE: Amicus - Learning from Pre-Roe to Navigate Post-Roe