That ain’t gonna do it. You’re going about it all wrong, and putting far too much work into it. What you need is to get hold of an Icelandic one-horned Einhyrningur ram, stick a whole bunch of colorful party streamers up its butt, and you’re done! Have it delivered with the pony, and consider our problems solved!
See, I gave them a couple of million, and it’s fixed in a few hours.
I couldn’t even get on the Boards until just now. I kept getting some weird error message (not a 50x) that said something about data errors. It said the “team” is aware of the issue and suggested I try rebooting the page or contacting someone. Neither link worked. But maybe, I thought hopefully, this means TPTB are finally fixing the issue!
Now I’m on–YAY!–but since TPTB aren’t communicating even with Tuba Diva, it’s hard to tell what that error message was about. I suddenly understand the desperation that leads people to see Jesus in a tortilla.
There was a message up from Tuba Diva earlier that it was going down this evening for maintenance, so the outage was expected. Since it came back, it’s quicker and no error messages for me.
I saw the database outage message and since then it’s been fine for the past hour or two. Based on past history, I’m skeptical that this is a permanent fix, but definitely something has changed for the better and let’s hope it stays that way.
Fingers crossed.
Actually that’s not the case.
They’ve been talking to me a lot about what’s been happening, and consulting with DesertNet.
They’ve been reading your posts. They are well aware of what you have been having to deal with here, especially these last few days.
One thing they got out of the discussions with DesertNet is this: the current software is one of the culprits here. (The other is the ginormous size of the database.)
Let me share this with you.
*There’s nothing wrong with our server, or the hardware, or the tech support keeping the current version alive — literally no improvements can happen until we get off vBulletin 3.8.3, because this old version of vBulletin requires a specific type of mySQL table that locks the entire database table anytime there are changes made to it, as opposed to a more modern version that only locks a specific row of the database. Any time traffic reaches a certain level and a great number of people are attempting to post at the same time (or automated internal house-cleaning processes kick in, such as the beginning of the month and/or weekend), it hits a bottleneck. *
That is exactly what you have been seeing.
*So we can attempt to upgrade to a newer version of vBulletin, but that just kicks the can down the road in terms of being locked into a proprietary platform with a history of intermittent updates. *
Going to an upgraded version of vBulletin necessitates making some changes. One of the results from these necessary changes would actually expand the size of the database even more than it is now; I’ve heard estimates that doing this would quadruple it. We’re having enough issues right now with 22 million posts.
TPTB have been trying to figure out what’s the best way to go here and also investigating how to answer questions such as:
What is the best platform for the site?
What offers all the things we need – size, speed, flexibility, serious customer support – at an affordable price?
How best to sustain this going forward?
And most of all: How to budget and pay for it.
These are all big issues and not decided easily or in a short period of time.
They are not playing with you. They are not ignoring you. They are attempting to make this come out right.
Currently the board does good to break even. This is with all the support costs and my stipend; I love you guys but there is no way I could do this for free. When you add in all the developer hours that have gone into this thus far, the Dope is a money loser for the Sun-Times. But here’s what they say about that:
… and we’re definitely losing money if you account for the developer hours that my staff spends on it. But that’s our problem, not theirs, so I’m not looking for sympathy.
*Either way, we have a lot of respect for the community and brand and want to do right by it, and a belief that if we can stabilize everything it’ll be a worthwhile investment for the Sun-Times to make above and beyond community good will. The board is an incredible piece of living Internet history and deserves to be preserved and a chance to grow. *
Everyone I’ve worked with on the Sun-Times staff thinks this site is meaningful and special and they want it to not only live but thrive. And they’re willing to do what they can within the confines of their business and their resources to make that happen.
If you’re serious about fund-raising, I have asked that they allow us to set something up, available to you through the subscription function already in place here. I also want to set up a “tip jar” or other form of donation on the other side of the site and get people reading the columns. I ask if you’re truly serious about this that you contact me off the board and give me some idea of how much money we’re talking about here – if I could come up with a meaningful figure that we could pull together --think of it as pledges, perhaps – that might help this snowball run down the mountain a little faster. Before you get all jacked up about that, please know that there may be legal or accounting issues that may keep this from happening. But I don’t yet have an answer for that. I have asked.
I also seriously want to do a podcast – have stuff in place for that already – but I would also like a sponsor for it. If you know anyone that would like to do this, please send me an email and we can surely talk about it.
Things are going to get better. Not instantly better. And hopefully what just happened tonight will make us more stable and the site more usable. But it’s only a band-aid on a sucking chest wound and to picture it as anything else is not useful.
Y’all have been so good. And everybody wants to make things better. Please bear with us and help us get there.
Thanks.
Jenny
your humble TubaDiva
Administrator
Thank you. “Band-aid on a sucking chest wound” is not very optimistic.
I’m not sure what you meant by the donations " on the other side of the site".
The content side, the columns. Different architecture, not sure if they’re even on the same server as the board.
I realize we have people that don’t even know we have columns. If you encounter these people on the board, please dispel their ignorance.
I may sound melodramatic there but I’m very respectful of what our current install of vBulletin has borne up under. It’s like 10 years old and we have beaten it like the proverbial rented mule. And it has valiantly tried its damndest to work for us until it can’t hardly rise up no more. That is yeoman service. It probably deserves a monument for “good and faithful servant.” While I would like to believe that the problems we have had are fixed by what happened tonight I have zero faith in that. (I’d be dee-lighted to be proven wrong.) We have the same problems; they just came up with something to get us through at least for now. If it holds it’s a blessing.
Jenny
your humble TubaDiva
Administrator
OK. Band-aid on a sucking chest wound it is. 
Tubadiva, that definitely sounds like a situation where it might be better to only bring over more recently active content, and leave the rest archived. (Maybe include a link at the top of each forum so that it’s still there.) We don’t need the vast majority of the site to be in the new database. That way the database doesn’t have to become all that large, even if the new system requires a larger database.
Of course, maybe there’s some forum software out there that can handle having active threads that need to be well-indexed and inactive threads that can be in an archive state until they get resurrected.
There’s also one other problem that just occurred to me–there are external links to the site, and it would be good if they didn’t break–if there was some way to keep the thread numbers and redirect to the new system.
Finally, if they do upgrade it, they’ll need to stay on longer to tweak the UI until they get it right. We can’t have a repeat of SultanTheme with parts of it broken–not once there is no option to use a classic theme.
Thanks for that detailed and informative post, TubaDiva. And things are much better this morning. I am no longer irked.*
*That was my five-letter word, but I enjoyed all the guesses.
It was, of course, an informative and encouraging post from Tuba Diva. But honestly - with this having been going on over two months, I feel someone needs to point out that it would have been helpful to post some of this background a month or more ago. Or were there such posts and I missed them? It greatly increases frustration when we have no idea what’s going on (or if anything is going on) behind the scenes.
I’m glad to know TPTB are finally accepting that VB3 is the problem. No doubt they are aware that VB4 is also past end-of-life and that VB5 is notoriously problematic.
This is a BFD. Joy to our small part of the world.
That’s all I needed to hear. That they acknowledge that we exist is cause for hope.
As to what software we ought to upgrade to, I don’t think anything but vBulletin will be able to import our database, so if you go with something else, moving all these threads will require a script to be developed and run. That will take man-hours and a lot of time to complete.
On the other hand, vB is a resource hog about some things, so moving to a slimmer, more modern platform might ultimately be our best chance for survival. As much as I hate the platform, we might have to go with something like Discourse just to stay afloat.
Tuba, thanks for the update. I really appreciate it. 
Greyson, I’ve posted about this before. vBulletin 4 is also end-of-life. vBulletin 5 is sloooooooooow. XenForo, Invision Power Board, and Woltlab Burning Board (the three other big commercial PHP/MySQL message board systems) can import vBulletin 3 databases out of the box. The importing process is pretty straight forward, although the conversion process will probably take a little while – a day or so.
I know XenForo 2 can automatically redirect from old vBulletin URLs to new XenForo URLs. For example, the URL of this thread (https://boards.straightdope.com/sdmb/showthread.php?t=890387) will redirect to something like https://boards.straightdope.com/threads/hello-mods-tubadiva.890387 .
XenForo (written by the same people who wrote vBulletin 3) and Woltlab are probably the fastest of the three.
Discourse is glorified chat.
This has been an extremely informative thread. Thank you, TubaDiva, for your heroic efforts and for keeping us informed! Based on my periodic visits to the board since since late last night when some magic was done to the database, there is no doubt that the problem has gone away. Previous experience suggests that the ongoing operation of the system will cause it to reappear, but let’s hope for the best. It’s extremely good news to know that TBTB care and are continuing to look at long-term solutions. 
Ah, ok. We’re on 3.8. That should work.
Agreed. Ugh. But it’s free glorified chat, though it would completely change the culture here.
Tuba Diva’s post was somewhat long- please don’t miss the part where she asks those of us inclined and able to support a fundraising to PM her with some idea as to what we, individually, could contribute to such.
My understanding is that we have been on 3.8.3 as long as we have because everything after that is a different database structure ; I also just learned it also takes working with a different flavor of mySQL. Conversion of our existing database to this new structure and all it takes to get there is apparently a huge freakin’ deal that greatly increases the size of the current database – so the pun on “huge deal” is intentional.
The IT guys for the Sun-Times did not normally handle vBulletin and they came in knowing nothing about it and having to learn as they went. Not something you get up in overnight.
What went before is not as important as what faces us going forward.
Things appear to be more better after last night’s optimization and I sincerely hope this holds us for a while until we can move on.
Jenny
your humble TubaDiva
Administrator