To clarify for the OP…
Windows provides an interface to the display - the GDI interface - which provides graphics primitives and some more complex functions. The windows manager operates here - the windows decorations and windows behaviour and mouse/keyboard is provided at this level.
Then there a some higher level shared libraries - buttons and lists and standard dialog boxes.
Finally, there is the Graphical shell - the default provided shell is explorer.exe. This handles the desktop and start menu and task bar, as well as disk exploration. explorer.exe runs in two modes - the first instance builds the desktop and taskbar, subsequent instances act as file explorers.
So - for a new windows interface, you can replace explorer.exe as a shell. This replaces file exploration, app launching, menus, toolbars etc. This is straightforward, but for compatibility you need to supply all the functionality that Explorer supplies - Shell Folders and environment and link creation and all sorts of other stuff.
You could conceivably replace all the standard dialog functions - maybe starting with the equivalents defined in the WINE (Windows on Linux) libraries, compiling your own DLLs and replacing the MS ones, or hooking the API interfaces. This could create new-look buttons and scroll bars, and change the dialog boxes etc.
Replacing the Window manager functionality in lower levels is far more difficult - you could only do this by hooking the API interfaces and replicating the functionality. This level would have to exactly replicate the existing functionality, or applications would break. I don’t think that anyone has done this, but I guess that the WINE and ReactOs (a free Windows style OS that can use Windows drivers) teams have had to build this level of functionality, so their code would be a start.
I personally am waiting for KDE 4 next year, because that should port to Windows quite quickly (as a native app and not as a local X-Windows Client app, which was done for older KDE enviroments), and I really like using it - but I don’t know whether they will implement a Windows Manager. KDE 4 sits on a standard set of libraries that works the same on Windows and Linux.
Si