Except that it was a huge success.
As to how the original worked, there were basically three kinds of content in the 3rd edition rulebooks. A few very specific bits of content (mostly creatures, plus the names of a few important NPCs like Mordenkainen and Tenser) that were first created for D&D specifically were what they call “product identity”. Off the top of my head, these include owlbears, displacer beasts, beholders, mindflayers, and umber hulks. The presence of these specific creatures definitively identify something as being based on D&D, and they’ve often used them as such, right from the start (for instance, the creatures on the cover of my 2nd edition Monster Manual were a beholder, a displacer beast, and an umber hulk, and at least three different 5th edition books have a beholder on the cover). They have a pretty strong trademark claim for these creatures.
Second, there the settings, descriptive text, and so on (other than the Product Identity things). These have a creative artistic element to them, and are pretty clearly copyrightable. If that was all they wanted to protect, ordinary copyright law would have been enough.
But third, there’s the mechanical rules. This is the largest share of what they created, and what they’d really like to protect, except that the courts have already decided that that sort of information isn’t copyrightable. So they needed to come up with some other way of protecting it.
Enter the OGL. Basically, what they did was they claimed copyright for the parts they could copyright, but then published a license to let anyone else who followed certain conditions to use those copyrighted parts (aside from the Brand Identity bits). And the conditions they chose were that it was licensed to anyone who agreed not to publish, share, or copy certain key pieces of the rules. So a person could, for instance, copy and share the XP by level table, if that’s all they do, because they can’t copyright that table… or a person can choose not to copy that one little bit, and in return get the right to copy all of the spells and their descriptions, and so on (except that the ones named after product-identity characters are slightly renamed). The OGL was such an attractive deal that almost everyone agreed to it, even though they had the option not to.
And yes, it did make it possible to make a game that was very, very similar to D&D, just with new tables to replace things like the XP table. Which is what Paizo did with Pathfinder, and so they lost percentage market share. But at the same time, it also made it possible for absolutely anyone to make things like new adventure modules, and even sourcebooks with new character classes and the like, that would be used with the books that WotC (or other gaming companies) sold, and so the hobby got a lot more popular, and WotC’s sales increased significantly even with the new competition from Paizo.