It does matter. Alan Krueger’s research on twins who attended different schools found that generally, the twin that attended the better school fared better in the labor market. This goes back to a pretty consistent finding in the research literature - peer effects matter. The smarter your classmates, the more rigor you will experience in the classroom as you compete.
Now, having said that, there are so many variables that factor in here. If you go to a state school with a great reputation in a certain field, people might be more impressed by that than an Ivy League degree (especially if that particular field is not one of their strengths). And what if you’re a standout at your university (class president, etc.)?
I did a prestigious honors major at UT. It would have helped if I wanted to go to med school or law school, but I taught after graduation and nobody gave a shit what I majored in; in fact, one of the human resource officers told me to stop using my major’s name and just say I was an honors humanities major. I then got my master’s and doctorate at Harvard (not at the same time; in my field the master’s can be a terminal degree). So I experienced the halo effect as a master’s graduate and as a doctoral graduate.
One of my professors had a saying: “No-one is agnostic about Harvard.” I think this is true. The second that someone mentions that Hippy went to Harvard, there’s an arched eyebrow. Sometimes it’s “oh, he’s gonna be smarter than average.” Sometimes it’s “oh, he’s gonna be a snob.” There was a column in Slate recently about the “H bomb” and the annoying tendency for Harvardians to mention that they went to school in the Boston area. The only people who say this are people who go to small schools that you wouldn’t know of unless you lived there (like Lesley College) or Harvard people. And truthfully, Harvard people are the worst perpetrators of this behavior.
The people who are generally indifferent are other Ivy League grads who aren’t insecure about their school. I know a guy who’s a Stanford Ph.D. who felt the need to tell me that he got into Harvard but chose not to go. Why would anyone feel the need to say that?
Hell, I went to Harvard for many reasons, one of which is the prestige. I will also say that for my doctorate, that was less important: I already had a Harvard degree and Harvard isn’t the top school for my specialty. I went because my mentor and other folks were encouraging me to come back and I knew the community well. I liked Harvard a lot, though I have to point out that I did not attend Harvard College, which is another experience altogether. In the Harvard pecking order, my grad program is near the bottom, with the College, Business, and Law occupying the top rungs. So no, my kids aren’t legacies, despite spending seven years in school there. However, the Harvard effect intensifies once you venture outside of Cambridge. I had a ton of capital on the job market and was offered both tenure-track positions I applied for. I like to think that I was great on the job talk, but I’m sure the Harvard halo kicked in too. To this day, when I’m introduced to other scholars by a colleague, they will often drop “he earned his doctorate at Harvard” in the message.
But I have lost count of the times that someone will tell me, “I can’t believe you’re so cool. I thought because you went to Harvard you’d be stuck up and use big words all the time.” Usually this is after a colossal screw up where it’s evident the Harvard guy doesn’t have his shit together. 