IMHO, college shouldn't be all about academics

Yes it is: the word “educate” covers networking.

I know what you mean, and I don’t disagree. There’s a lot of tacit bias in education that basically throws away a lot of great minds because they don’t conform to the mould early and often. But we do that in other sectors, too: jobs, sports… it’s a widespread cultural failure, not just in education.

And snobbery and elitism are pretty widespread human failures, too.

A company hiring people who has limited resources is going to bet that the average student at Harvard is going to work out better than the average student at Podunk U. BC is a good school, so I’ll leave it out of this.) That’s valuable right there.
Top schools also give you access to more famous professors which is valuable if you get interested in an area. The level of instruction is higher thanks to students catching on faster. I’ve taught classes at a top school and not so top school so I can testify to this.
The US is relatively good at not targeting kids. My son-in-law is from Germany, and he didn’t really work at age 13 and 14 and got assigned to a vocational track. He got serious and had to go to a secretarial college to qualify for going to a real college. He eventually got a Masters and is doing quite nicely. I appreciate that it is hard to break out of low expectations here, but not that hard.