The switchover occurred for more down-to-earth reasons than the rise of auteur theory, though that did play its part. The change was a result of the collapse of the studio system during the 60s and 70s, which radically weakened the creative role of the producer.
Back during the heyday of the studio system in the 1930s and 40s it was the producers who shaped and molded pictures: they not only picked the director, the material and the cast, but they also often supervised minutae such as costumes, wardrobe, and script details. The director was considered just one more cog in the machine–a talented cogs to be sure–and the machine was tended by the producer.
The most famous producer of this era was David O. Selznick, but there were many others who fit the role of the creative producer–Hal Wallis at Warner Brothers, for example.
The Best Picture Award is given to the producer. During the creative producer era, movies that were dominated by the producer tended to win Best Picture; those that were dominated by the director would win Best Director. Take 1940, for example: Selznick won Best Picture for Rebecca (the closest Hitchcock ever got to a non-honorary Oscar), while John Ford won for The Grapes of Wrath.
But from 1945-1964 the studio system fell apart for a variety of reasons, as stars and directors found it both easier and more profitable to form their own production companies. As the the studio system collapsed and stars and directors gained more power, the producer’s role shrunk accordingly: they were no longer creative supervisors, but just money men who managed the financial end of the project.
What you often see nowadays is the stars and/or the directors acting as producers along with a couple of more technically/financially oriented people. E.g. you’ll look at the credits and see “PRODUCED BY: Big Name Star, Big Name Director, and (two guys you’ve never heard of).”
So Best Director/Best Picture splits have become rarer, both because it’s far more common for directors to also be producers now than it was in the 30s and 40s (e.g. LOTR: Jackson is one of the producers) and because the creative producer has become virtually extinct … with, for better or worse, one prominent exception–Jerry Bruckheimer.