The message boards are going to be taken down tonight (Friday, June 14th) starting at 1 AM CDT. The master database is going to be taken out of service to be thoroughly checked and possibly repaired. While that is occurring the message board will be unavailable. Expected downtime is 4 to 6 hours so the message board will not be available again until some time on Friday morning.
This is the first step in a series of steps we are taking to address the current functionality issues. While tonight the master database is being worked on there are multiple replicated slave databases used by the system as well. They will also need to be checked and possibly repaired. Those checks can occur while the message board remains online and will occur from Friday night throughout the weekend. While any of the slave databases are offline there will be some degradation in performance.
In tandem to this work on the underlying vBulletin databases we’ll be rebuilding the various internal vBulletin caches and indices. Due to the size of the database this process can take quite a while and I do not have an estimate yet on when that work will be completed. It will be in the range of days though not hours.
At this point we have not determined the root cause of the functionality problems so it’s unknown if tonight’s upcoming database work will lead to some resolutions of the issues. It is a first step in a process that will eventually lead to a resolution.
Finally, waiting in the wings is a new front end web server to replace what is currently being used. Right now the front end server is on 3 year old hardware. The new server is a current operating system build and new software stack that should provide performance benefits to what we see today. Testing is ongoing. It will be installed sometime in the next few weeks. As for the back end database we believe it has more than enough horsepower. The SDMB database shares resources on a 64 core 5 server database cluster running with SSD storage technology. The SDMB database usage averages about 4 cores of utilization under normal load and can peak up to 11 cores so there is of headroom on the back end.
Hamsters galore! We shall see if this solves (or re-solves) message board problems, world hunger, and everything!
And as long as some capital is being thrown at new hardware, is there also any thought given to upgrading to a newer version of vBulletin? (And does anyone know of any arguments why a newer vBulletin is better? I know nothing about that.)
Newer is not automatically better and being an early adopter is not always the best course of action. In dealing with these issues, the opposite seems to be the case; we have been much better off by being the latest adopters of all over the years.
There’s some evidence that going to a newer version of vBulletin makes for a slower database. Apparently moving past the current version we’re on changes how the database operates and not in a good way, makes it harder to slog through the records and etc. The Straight Dope contains an immense database and anything that would make that more difficult to navigate or put more strain on the system is a lousy idea.
Thank you!!! The other messages made it sound like you guys were just going to a new server without figuring out the problem. I see that was an incorrect interpretation.