Was the pre-1990 Internet completely unsearchable?

Factual Questions
1

Was it possible to even do a keyboard search or send out a bot if you were knowledgeable enough? Or was there absolutely no way to navigate newsgroups, FTP etc before Archie and Gopher came out? I mean aside from human-made menus of course.

Apparently net bots were patented as early as 1986, though I’m not sure if they were actually used that far back. The earliest articles I was able to find describing their use refer to the KIS bots which was apparently patented in 1988 but the article dates to May 1990.

Was the WHOIS tool a genuine search engine in the early days? I’d be surprised if nobody had the insight to think of a way to automatically index the Internet until 1990, that sounds pretty late even if it was still pretty niche.

2

Actually, it might have been unsearchable prior to 1990.

Archie was used to search FTP. It was the first search engine on the Internet, and it dates to 1990.

More.

WHOIS isn’t really a search engine protocol. It was (and is) a directory service, tying domain names to people and organizations, more like a phone book than anything else.

3

Since I hate editing under the Sword of Timeout (hi, Indistinguishable!): The Internet prior to 1990 was small and extraordinarily ad hoc. The Domain Name System as we know it, which automatically maps host names to IP addresses, didn’t even exist until 1983; prior to that, the only way to reach a system was to either have received a version of the HOSTS.TXT file with that system’s host name in it or to know its IP address. Yes, people were emailing around a hand-edited hosts file. Yes, that worked.

In fact, it wasn’t always clear that the Internet would become important: The OSI Protocol Suite, including such hits as X.400 email and X.25 packet-switching WANs, was a contender for a good long time until the Internet played the nasty trick of becoming useful faster and outgrew the Big Serious Official International Standard Network. People still try to shoehorn the Internet into the OSI protocol stack, which must be right because it’s so much more complicated than necessary!

Anyway, digressions aside, my point is that search engines only become a necessity when there’s more information out there than people can find by navigating directory trees on a handful of hosts, and the Internet didn’t hit its growth phase until the later 1980s at the very earliest; really, it didn’t seriously take off until the 1990s, when the Web hit the mainstream.

4

Was KIS more similar to WHOIS or to Archie, or somewhere in between?

5

Actually to my understanding it was already growing exponentially well before 1990, though granted from a tiny base (maybe a couple thousand users?) in the early 1980s. Still, by 1988 there were thousands of networks on the Internet if you count the UUCP, Fidonet, etc.

There were already 60,000 hosts online in 1988, which is about the number of Web servers that existed in 1995 so while it’s fairly small potatoes compared to even a few years later we’re already talking about a large network.

And of course you could make the argument that the Internet is only IP/Arpanet/NSFNET material, but I’d make the case that’s a bit overly simplistic since the newsgroups, BITNET and so on could feed into them to some extent.

What makes something a “search engine”? To my understanding, Archie just pinged hosts and waited for them to approve and then listed them. I’d argue that’s not really a true search engine in the way we use the term today. It’s more akin to a phone book than to Google in terms of how it functions.

6

Is this what you’re talking about?

Intelligent agents were a good idea that basically went nowhere until software like Siri and Cortana took off. They were meant to be digital factotums, doing a whole raft of things based on some limited AI and a whole lot of information.

I could probably come up with reasons they never really took off, from the fact intermittent dial-up Internet access is the enemy of a bot which really needs to constantly monitor remote services to do its job to the fact you have to leave your computer on all the time to get scheduled tasks to run at the right moments, but the big reason was probably simply that the intelligent agents which actually existed didn’t provide a lot of very compelling services in the 1990s. A good Web portal could give you a news summary (and they all did) and online shopping, another thing intelligent agents were supposed to do for you, was done well enough by hand.

So intelligent agents weren’t quite search engines because they weren’t quite like anything we ever actually had, then or now; we’ve come a lot closer, though.

7

UUCP and Fido weren’t the Internet. They were different networks with different goals, users, and technology. Their traffic was intermittently gated into and out of the Internet, but that doesn’t make them the same thing.

This looks like a search engine to me. A search engine must at least provide the ability to do text searches of file names. That’s apparently all Archie did.

If Archie doesn’t qualify, and I agree it’s borderline, then the title of “first Internet search engine” must go to something developed after 1990, so the answer to your question in the title is still “yes”.

8

I remember in the early days of the WWW, we had a web page that listed every other web page on the internet. The entire list was only a couple of pages long. You want to search “the internet”? Just scroll down through the list.

FTP and e-mail were the big things back then. Companies and individual people would put things on ftp sites, which was a big step up from the dial-up bulletin-board systems before then. With a BBS, you had to know what number to dial. Computer magazines would have some listings. Local computer clubs would have others. Sometimes things went around by word of mouth. FTP sites were a lot of the same. You found out about them from magazines, local clubs and user groups, and word of mouth.

One thing that helped was Usenet groups. If you were interested in something, there was probably a usenet group that discussed it, and you could often find listings of ftp sites and such in that group’s messages. Back in the day, I kept a list of electronic hobbyist FTP and web sites, as well as the snail mail addresses of companies that you could mail order catalogs and parts from. It started out as a fairly short list, but grew pretty large in the early to mid 1990s. Periodically, I would publish this list in a particular usenet group related to electronics hobbyists.

The way that usenet worked was a lot different than the way a modern message board works. With a modern message board (like the SDMB), you have one sever and everyone accesses that. With usenet, every computer system that wanted to access it subscribed to it, and then that computer system received every message that went out on usenet. So you didn’t have to search for message boards, they were all there under usenet, and every system that accessed usenet had its own local copy of every single message. So if you posted something on usenet about your favorite musical group, that message got copied to every single usenet archive all over the internet. It worked that way because it was really built to send out news and messages to just a handful of computers in the early days of the internet. General purpose message boards had never been a part of its original intent.

Creating newsgroups back then also required approval, and was fairly difficult to do, with one exception, groups in the alt section of usenet. Anyone could create an alt newsgroup (there was an alt.fan.cecil-adams group at some point). Since anyone could make an alt group, this is where the general purpose and popular newsgroups often ended up. Also, because of its mostly unregulated nature, as well as its higher level of traffic, some computer administrators would choose not to accept messages from the alt hierarchy, and their usenet archives would have no alt newsgroups in them.

Searching for newsgroups was easy. You just browsed through the group names. Whatever computer system you were accessing usenet from had the entire list of groups, so you didn’t have to search multiple computers. Every group and every message was right there.

9

It was like this. It wasn’t so much ‘searching’ as ‘looking up’. But then there wasn’t that much to search for either.

10

In that case, are Netflix and HTML-5 based phone apps not part of the Web? They link to the Web to some extent and can be accessed via the Web, but they are in some senses separate from it. I guess it depends on your perspective. To me the Internet is not just TCP/IP or whatever sprung off Arpanet/NSFNET/Alternet but would include all the other networks that gateway to it. Though I admit that’s a subjective opinion that could be argued either way.

11

Interesting. So it wasn’t so much that it was impossible to search, but rather unnecessary due to the limited scope of cyberspace at the time.

12

You’re trying to group unlike things.

HTML5 is a standard to which Web pages can be written. It offers ways for pages to request browsers to do certain things in ways modern browsers are likely to understand. HTML5 web pages are delivered using HTTP, so they’re undeniably part of the Web.

Netflix is a company which offers videos to watch. Those videos can be accessed using multiple technologies, one of which is a website which likely uses HTML5 functionality. That specific delivery mechanism is part of the Web.

When I access Netflix’s stuff via my Roku, I’m using an app written in Brightscript, Roku’s Not-Visual-Basic programming language. That, as far as I can tell, isn’t using HTTP, so it is not part of the Web. I might be wrong, it might be accessing the videos using a TCP connection over which HTTP is being spoken, but as far as I can tell, it isn’t.

13

A list of all web pages doesn’t do you very much good for searching the Internet, unless one of those web pages on the list was some other search tool. The Internet is and always has been a lot more than the World Wide Web. Nowadays, the Web is the largest publicly-visible portion of the Internet, so it’s easy to conflate the two, but at the time when a list of all Web pages was only a couple of pages long, that wasn’t the case.

14

Be careful, that some people might be thinking that “The Web” and “The Internet” are synonymous.

“The Web” (if, by that we mean all those on-line communications using HTTP) is just a subset of all the stuff that happens on “The Internet”.

“The Internet” consists of a whole universe of servers, client machines, and all the related programs running on servers and clients, and all the hardware and software and protocols by which they all transmit information among them. “The Web”, as I understand the term, specifically refers to the use of the HTTP protocol for transmitting the information that we commonly call “web pages”. Of course, there is a lot more than that going on out there.

ETA: Wow. Partially ninja’d by Chronos there.

15

It wasn’t all that hard, since business cards with email addresses gave paths from {cbvax, decvax,ihnp4}. So there was usually only one or two hops.

The closes thing I can think of to searching usenet was search for the groups in your .newsrc file, and then maybe searching within loaded posts.

(I just checked and I still have a .newsrc file in ~ from when I still read news, about 15 years ago. )

16

Search as we know it was invented by Google. Prior to Google, website owners had to send their website URL to Yahoo, Alta Vista, etc. individually, along with a list of keywords associated with the site’s content. If you typed in one or more of those keywords, you got a hit, but otherwise, you were SOL. Most of the time, I learned about websites through word of mouth or they were written down in an advertisement or something. Search before Google was a different beast entirely. It didn’t even work as well as “ctrl-F” does in Word.

17

Pretty much… I started college in the fall of 1991 as a computer science major, so this was something I was interested in.

Back then, you had gopher sites, Archie searching, Usenet and FTP sites. With the exception of Usenet and a few Gopher sites, most of the rest of it was pretty local stuff, and centered around your particular university’s departments. There just wasn’t much to look at on other sites, unless you liked looking at the class schedules and the like at other schools. For the most part, Usenet and various sorts of online chats and MUDs were where the action was.

By about 1993, the WWW had become a thing, sort of. There were sites, and they were hypertext, and some even had graphics. Most sites were personal-interest type sites, hobbyist sites and the like; you could go look up Babylon 5 stuff on the Lurker’s Guide, and look on IMDB. Still, by the end of 1993, there were less than 700 websites total. By the end of 1994, there were upwards of 10,000.

Lycos was the first major web search engine in 1994, followed closely by Altavista. Google didn’t come around until a few years later in 1998, but it took it a while to become dominant.

18

I had a book (THICK book) from around 1995 that was sort of like yellow pages. Sorted by catagory, with a list of websites pertaining to each catagory.

19

Somehow, for some reason, the Netscape Welcome Site is still up and running. :eek:

It looks like they stopped updating it around October 1994, but it’s all still there (lots and lots of dead links though - as you’d expect from a 21 year old site!) and is a fascinating snapshot of how the web used to be.
You wouldn’t search for sites so much as browse for them. Check out the ‘what’s new’ and ‘what’s cool’ sections - I remember that was the way you traveled the web back then. Search engines were still at the point where they were more annoying than useful, so either directory searches or web portals like the Netscape one here were the order of the day.

20

Wow. Click on “Internet Search” and you’ll see some old ones. Apparently “World Wide Web Worm” was a crawler of the “ctrl-F” variety.

8??