An OS is not a specific user interface (UI). An OS is a way for application software (browsers, text editors, the kind of things you start up to do something specific) to speak with the BIOS (Basic Input-Output System, which hands the commands directly to the hardware). OSes also manage things like keeping the files in a usable way (via a filesystem of some kind) and communicating with the outside world (from managing keyboard and mouse interrupts to handling peripherals and network connections). Finally, OSes can handle programs as processes, making multitasking possible in some OSes (it’s always been possible in UNIX and UNIX-based systems, like Linux, but it is not possible in any DOS or older versions of MacOS).
Case in point: Linux is, by default, text-only. Like UNIX before it (and still today), the basic way to work with Linux is through a command line. You can only get a GUI (windows and such) if you start up a seperate application, usually the X Windowing System, but you can access everything Linux is capable of (short of rendering graphics) from the command line interface. X is an application that manages the basics of having a GUI and translates things like mouse clicks into data programs can understand.
While in that GUI, you can open a terminal emulation program and get a perfectly usable command line. You can also accomplish things with points and clicks, depending on which desktop manager (the program that determines what your GUI will look like and how it will behave) you have running and how that program is configured.
Macintosh (pre-MacOS X, which is a UNIX variant) and MS-Windows and Amiga have all intertwined their GUI with their OS’s basic features to the point where running a text-only session entails running an emulation of a CLI (Command-Line Interface). The MS-DOS box WindowsXP features is a program that emulates a (nonexistent*) version of MS-DOS.
*`nonexistent’ in that it only exists in that emulation, not as a stand-alone OS.
You want a discussion of nonstandard UIs. OK, we’ve given you a few (with the Amiga’s being one of the most interesting, IMHO). Do you consider stylus-based interfaces seperate from GUIs? If so, look at various PDAs and cell phones for a cross-section of that breed. How about voice-recognition programs that allow the blind to operate computers? Or maybe you could look into VR systems, although I don’t know how much serious research has been done on making VR systems viable replacements for traditional UIs.
“In the Beginning was the Command Line” by Neal Stephenson – A great treatment of the whole subject of OSes and general computer history, and an essential read if you want to understand the psychology behind all of these machines and market forces. (It’s dated in spots: It referrs to the MS v DoJ trial (now concluded) and Be, Inc. as a living entity (now bankrupt).)