The point is, there are multiple decoupled layers to a Linux UI
The kernel, which provides the interface to the hardware and an environment to execute programs
The X display server, which is responsible for displaying graphics primitives and glyphs on screen. Any X client can display on a screen via an X Server.
The Window Manager sits between applications and the X server, and provides management of multiple display windows and the mouse/keyboard. The WM manages things like tiling, overlapping, focus, switching and window decorations. These are the basic functions of any WM. Advanced WMs may also provide animations and fancy virtual desktop switching (i.e Compiz). A WM should manage any X Client, but some clients may request advanced features that another WM may not provide. YMMV.
The Desktop Environment may (or may not) provide a set of applications with common infrastructure/look and feel/configuration/high level libraries. This includes things like Gnome, KDE, and CDE. Gnome and KDE use high-level toolkits (GTK+ and QT) to simplify programming, and have common architectures for ease of programming. As time has progressed, some features have moved out of specific desktop environments to a more generic shared common base (i.e D-Bus has replaced Bonobo from Gnome and DCOP from KDE). This has improved the ability to run multiple environments on a linux distribution.
X Clients are applications. A pure X application can run on any X Server using any Windows Manager. A QT or GTK application can run on an X Server with the appropriate toolkits, but does not necessarily rely on Gnome/KDE. Gnome/KDE apps are highly integrated and require the entire Desktop Environment or suitable shared components. You can easily run Gnome apps in KDE or vice versa, but additional subsystems may be started to support the app. In many distributions, applications (like Firefox, say) are modified to fit the distribution Desktop Environment.
Oh, and while X does use sockets to communicate, for local X-client_to_X-server connections it uses a file stream socket and not a TCP internet socket - however, using internet sockets does allow X clients to be completely decoupled from the X Server. Traditional X is stateful, though, and does not allow session teleportation from server to server. VNC (a X based remote screen access protocol) was designed to allow stateless remote access, and thus session teleportation to any screen.
There is also more to a distribution than just the UI/Visual shell. While file system layouts have generally achieved a level of commonality in recent years, there are still differences in how distributions manage packages/application installs, starting and running system services, and security subsystems. You could turn a Fedora system into an Ubuntu clone by copying Gnome themes and menus, and installing a specific set of packages, but the level of integration that comes with a full distribution will probably not be there.
Si