one problem I see with making the archive a premium product is that it may lead to much more thread repetition. Right now, many users (most hopefully) will search the archives before starting a new thread to see if the topic has been discussed before. If the archive is pay-per-read, this won’t be possible for those not paying for it, and thus many more repeat topics to wade through.
Well, I would bet a lot of people actually don’t search the archives. But then again, this isn’t always necessary.
For example, when starting a new thread in any of the column forums or ATMB, one should probably do a search to make sure the topic hasn’t been covered recently. Same with GQ and GD although one should in that case search a little further back than ATMB (maybe up to a year?). But MPSIMS, the Pit, IMHO, and even Cafe Society to a degree don’t need to be searched before a new thread is created, IMO, because the threads there are generally looking for opinions or an opportunity to rant (or babble), rather than discussing a topic or a question.
And of course, as has been noted, using the search engine does put a strain on the server, no doubt. So putting the archived material somewhere else, whether it’s another server or a CDROM, would lessen the load.
How about changing nothing? It may be painfully slow at times, but that’s part of the board’s unique charm. And any sort of fee-based system, or members-plus system, is just going to screw up membership. Is a faster server worth the risk of ruining the membership, the contributors who make the board worthwhile in the first place?
It’s not just that it’s slow… It’s also unstable. There have been numerous database crashes, and data lost on more than one occasion. It looks to me like the SDMB isn’t going to hang together forever the way things are going. And if would be a tragedy if this database were lost.
That kinda sounds like the basis of conservative ideology. IOW, “Things are fine now, if you try and improve them you’re might ruin what we’ve got.” I guess the liberals among us would reply, “Nothing ventured, nothing gained.”
Personally, I am open to many changes to the board. vB Polls and Avatars especially. And, I’d be willing to contribute to the bandwidth and hardware costs in many of the schemes suggested. The SDMB is unique as far as I can tell, a message board devoted to the truth. I’d pay $20 a year to make it more useable.
- Avatars take up a lot of bandwidth, compared to most features.
- In vBulletin’s current incarnation, the only way to turn avatars on or off for a group of people at once is on the basis of post count. So
- The SDMB can’t make them available to everyone without further bogging down the board, and
- It can’t make them part of a premium-membership package, period.
Unless somebody’s got a way around this, we should really ditch the subject of avatars.
If I read them right, ataraxy, I don’t think Mtgman, Sam, or dan are proposing pay-only access to old threads. I think they’re suggesting that the CR copy its archives onto CD-ROM and sell them to those of us who have sufficient interest and money to buy them. This would allow us to search old SDMB threads without putting additional load on the server - in fact, without even being online. I’ve got to say I like that idea a lot.
I wonder (a) how many CD-Roms it would take to contain the board archives from 1999-2001, say, and (b) how much it would cost the CR upfront to put that first copy of the archives onto CD-ROM. (After that, I’m assuming per-unit costs would be slight.)
RexDart - I agree that if the SDMB requires payment to post, it will hurt the boards. But we’ve bandied about a number of ideas in this thread by which the SDMB could potentially create revenue without requiring anything of anybody to maintain their existing level of access to the board. I can’t see any of that harming the SDMB.
And while some may find the board’s slowness charming, I suspect that most just find it frustrating.
And there’s one other thing: if the CR were to have a revenue stream directly attributable to the SDMB, then it would probably make them a lot more reluctant to pull the plug on it, should there ever be a reason to do so. I have no reason to believe that they’re contemplating such a measure, but you never know what the future might hold, and the time to insure oneself is before disaster strikes.
The database is neither “not all that big”, nor text. Are you talking about something else? 
The “archive” is a series of MySql database tables, organized in a database. It is not text, and to view it properly, the easiest way to do it is that you need:
- A webserver
- PHP
- MySql
and a machine set up and configured to use all of these programs. This is not a simple thing that could be distributed on a CD.
jdavis at one time said that the entire database was well more than 800-900 MB. I imagine that it is quite a bit larger than that by now. To put things in perspective, my database, including the Search indexes, which is only 49,151 posts, takes up 82 MB. It compresses down a bit on zipping, but still…you can’t run it zipped.
There is some common overhead, but let me tell you two stats. The 49,151 posts take up 29 MB in the “post.MYD” table. The two Search index tables take up 35 MB. Why are they more than the post table? Damned if I know.
This would not be hard to do the first time. The first time, what you would do is copy the entire database to another server, then delete the old articles from the current “active” database. You then would need to run some MySql optimization routines to reclaim the space and defragment the database tables, and it would take oodles and oodles of CPU power. Most likely, the database would have to be “offline” for days, or longer.
The problem which comes in is when you want to move articles (threads and posts) from one database to another, as part of a quarterly housecleaning. You would have to write some sort of serious export script, and a corresponding import script, and God help you if you screw up one little thing, as you may as well delete everything and recover from the nightly backups…
Also - all thread links to other posts/threads/whatever would be broken, as the threads and posts would no longer exist.
There are other alternatives.
My favorite one is making it such that you must pay to post in IMHO, MPSIMS, Cafe Society, and the BBQ Pit. Ignorance is still fought, in that people are free to post in the Comments on Cecil’s Columns, Comments on Staff Reports, GQ, and GD.
Some have worried that people would just “pollute” the free forums with material that belongs in the paying ones, to get around it. And that this would make the “hard-working Moderator’s job much harder”.
No it wouldn’t. One’s job in moderating misbehavior is as hard as one makes it. Give Moderators the right to ban, and let them ban as-needed. If really wanted, give the Banned person the right to appeal to the Admins, or make a “three strikes” rule, or something. Like people say, it’s a free net, there are upteen million other places to go, and if someone consistantly won’t follow the rules and is an almost always an asshole, and has nothing really to contribute to the “core” forums, why do they seem to be unbannable?
I could expound upon this, but it’s not the point of this thread.
And don’t forget the $85 a year vBulletin license, which is absolutely required for this, you dumbassed Una.
The short of it - it is entirely technically infeasible for 99.99% of the Members here. And that’s not good market penetration.
Anthracite, I completely neglected to consider the state in which the database currently exists.
How is this an alternative? People would still need to pay, and as the quoted post implied (I think), requiring payment to post (whether in some forums or all of them) would relfexively turn off a lot of people. I don’t see how this is a plus, even if the remaining people post only Fighting Ignorance topics.
Did I misunderstand?
On a top end server, vBulletin will begin to crawl at about 500 simultaneous users, and max out at 1,000.
vBulletin has been modded by a few board operators to use Oracle and Sybase.
vBulletin can only use a MySQL-based database. MySQL is reliable and cheap (as in free), but it’s a dog. The vBulletin implementation of a PostgreSQL back end is still six months to a year away. PostgreSQL is much faster then MySQL, and it’s also free.
Maybe a database guru here can donate not money, but time … time to come up with a hack so vBulletin can use a PostgreSQL back end.
As a last resort, there’s always the vBulletin clones that can use other database back ends. Copy the vBulletin to … oh, InvisionBoard, test it out, and see if if works. Let folks compare the two.
I think so. I’m saying that anyone could register, and be able to post in the comments Forums, GQ, and GD. No fee required.
If someone pays their $20 a year, they get a Member account that lets them also post in IMHO, MPSIMS, Cafe Society, and the Pit.
It would not limit or restrict the person who comes here to ask a question of the knowledgebase here, to pose a debate, or to comment on Cecil’s columns and staff reports.
Looks like that kills the archives-on=CD issue.
The size wasn’t exactly intimidating - with a CD-Rom holding 700 MB, an archive would fit on a few CDs. I was worried that it might be a few dozen.
But you’re right, Anthracite, not a whole lot of us are going to load up MySql, vBulletin, and all the rest, even if the $85 for the vBulletin license doesn’t faze us.
But elmwood, what you had to say is the more worrisome bit.
If vBulletin starts getting cranky with 500 users at once, and grinds to a stop at 1000, then even if the CR got us a new server and adequate bandwidth, the SDMB would soon be a victim of its own success, as the inevitable increased use caused us to run into vB’s scalability issues.
Until there’s reasonably user-friendly MB software that doesn’t have such a problem with scalability, it sounds as if we’re looking at a more fundamental problem than generating revenue for this place.
Bummer.
Anthracite: The archive may be in MySQL now, but it doesn’t have to be.
The reason the index is larger than the text is because full-text searching requires storing a reverse index to every word in the database, which is almost always larger than the word itself. I wrote software to do full-text indexing for years, and know this field extremely well.
And just because the database is sitting in MySQL now doesn’t mean it HAS to be. If I wanted to make it a product, I’d export the table data into another format made just for searching. I’ve done plenty of this kind of work in the past. Sure, it requires a pro who knows what he’s doing, but it’s not rocket science unless the database is encrypted or in some extremely strange format.
Or, just export it to another relational database format like btrieve which is available as a linkable COM object, and write a search front end that contains the database code natively. No install required.
My major concern is that this database is not lost. I think a lot of people underestimate the value of the SDMB database. There is the collected wisdom of an awful lot of extremely intelligent people stored in that thing. As a research tool, it’s invaluable. As a snapshot into a wide range of cultural and social issues, it’s invaluable. As a personal diary for a lot of posters, it’s valuable. I’m sure the reader is doing backups and all, but like I said, data has been lost on several occasions, and that really worries me. Plus, if the reader ever decides to pull the plug on the SDMB, I would hate for them to just trash the database or lock it away on some tape to be lost or damaged or forgotten.
At the very least, I hope they are burning regular copies of it onto CD-ROM and doing proper off-site backups. Twenty years from now, I really hope to be able to go back and read what I was saying when I was 20 years younger. When people want to know about obscure pop culture things from the 2000’s, I hope the SDMB database is around as a reference source.
Imagine if this archive went back to 1980. How valuable would it be to us to be able to go back and revisit the archives and read what we were saying about Reagan’s election as it happened? That’s the situation we’ll be in in 20 years, IF this database is protected.
And it seems to me the best protection would be to distribute it. I would be perfectly willing to volunteer my expertise and time to such a project, whether it means distributing it on CD, or building a new searchable front-end on another server if the Reader doesn’t want to let the database out of their hands. A lovely side benefit is that there should be lots of ways to turn it into a revenue source for the Reader and lower the workload on the main SDMB server at the same time.
But unfortunately, having to pay to post in the “fluff” columns would definitely turn off a lot of people (if the posts I’ve read in the past are any indication). Whether it’s the IMHO forums or the GQ forums that are pay isn’t relevant; some people will be alienated by the rate in any event.
It sounds as if you’re proposing that people could not post in IMHO, MPSIMS, CS, and the Pit without having that $20 membership. Am I reading you correctly? 
But unfortunately, having to pay to post in the “fluff” columns would definitely turn off a lot of people (if the posts I’ve read in the past are any indication). Whether it’s the IMHO forums or the GQ forums that are pay isn’t relevant; some people will be alienated by the rate in any event.
It sounds as if you’re proposing that people could not post in IMHO, MPSIMS, CS, and the Pit without having that $20 membership. Am I reading you correctly?
I’m sure any number of people might be willing to do it, but the CR is loathe to make any changes, let alone a hack…
FWIW, I was not the first one to come up with this - I believe a couple of Staff members proposed this in a past thread.
Quite simply, one way or another, it seems that the Board will need money. And if that money is to be obtained by Member fees, there are really two alternatives:
-
Every man, woman, and child pays a fee to post anywhere.
-
You only have to pay to post in the “non-core” Forums.
If it comes down to these two alternatives, I think you would agree that option 2 is the better one. However, the question really is - does it come down to only these two alternatives?
There have been numerous and plenty sources proposed for funding the Board. They all have been either rejected out of hand, or have been under consideration. I still do not understand why some of them have been rejected, but I no longer care. There obviously is a reason that they feel makes a good business case not to implement any of them, and I’m not being paid for management consulting for the Reader so I don’t know the details. I have my own house to take care of.
Well, as far as we uninvolved, not-mattering people are concerned, those aren’t the only two alternatives. And since none of us in this discussion carries any sort of weight with the CR, all alternatives are available to us. As far as we’re concerned, there are a lot of alternatives for revenue streams.
But if it does boil down to those two options, then option 2 is the better one.
Geez, what a dumb post by me. What I meant to say was that whereas there are plenty of alternatives available in our eyes, the CR probably (and presumably) doesn’t think so.