Ah ha! Thanks.
When things don’t work correctly here we have to contact Jerry. None of the admins here have server-side control and most all board functionality is controlled from the server side. We can tweak a little but that’s it.
I also note that it’s Sunday. Even Jerry gets time off.
:smack: Sorry! :smack:
I have the same request.
One would think even 131452 seaches/second; that is every member searching every 30 seconds 24/7 is a manageble number. And I don’t think the actual number is anywhere near. So please explain how to ‘abuse’ the search system.
If, in the very likely event I don’t understand jack about how straining searching is for the servers, **how about a limited number of searches/member. 30 searches/day would cover all my searching needs **
Has any analysis been done on the ratio of “legitimate searches” to “New Post browsing?”
If a lot of dopers are using the New Posts feature every time they finish reading a thread, I can see that this could become a problem. In fact, I have been guilty of this myself.
Maybe instead of restricting searches to every 2 minutes, you could try disabling the New Posts feature (at least temporarily) to see if the problem goes away.
I know this would make it a little more inconvenient for those who are used to browsing this way, but if it speeds up the board it may be worth it.
I use the New Posts function myself a lot and would not want to see it go away.
Ditto.
Is “New Posts” usage actually the problem? If so, then I too am guilty :o – but I think the system really should be built so that it can handle this kind of load. If not, is there any way of making the minimum interval between New Posts’ invocations different from the minimum interval between key-word searches?
The problem is that searching basically locks up the database until the search is completed. Normally this takes a couple minutes during which no one can post, a relatively minor annoyance. On Saturday the volume of searches was such that the board went down for several hours. No sooner had Jerry gotten the board restored than a single individual proceeded to submit 8 searches in 4 minutes, dropping the board again. Taking the view that an operating board with 2 minutes between searches was better than no board at all, Jerry increased the interval. If he seemed a little cranky, that may be because it was a nice day and he hoped to be out planting broccoli rather than staring at a computer screen.
Please realize that this problem is not hardware-related; it’s inherent in the design of the bulletin board software, an otherwise pretty reliable commercial product. Please realize also that having search crash the board is unusual - although we suspected vandalism at first, we’ve since learned a thread generating a lot of searches was in progress around that time. To avoid future disruptions, we ask members to be judicious in their use of the search function.
Mr. Zotti, any thoughts at all of taking up the many offers to possibly improve search functions. We have a large number of IT Professionals on this board. Maybe you and the staff could choose a group of volunteers and provide them with the information to design a better search engine for the Dope.
Many have offered in this thread and via Emails, PMs, possibly a hidden forum, maybe a group could create an excellent, secure and free solution available.
Jim
You mean, if any single member does a search, he (or she) locks out all others who wish to post at the same time? That just seems bizarre. It sorta sounds like The Reader isn’t giving Jerry the tools he needs to do the job.
Long ago, and frequently, I suggested that searching was the culprit causing the holdups on an otherwise fast server. Now it seems my suspicions have been confirmed.
Two things come to mind. First, if there is any setting that users can use to mimimize the holdup, I sure would like to know about it. If I can specify searching for one week rather than one year and save processsing time, I would certainly try to help.
Second, it seems unlikely that SDMB will want to hack the vBulletin software for all the reasons that have been advanced in the past. But we can’t be the only BB that is affected, so I think vB needs to be contacted and encouraged to make search improvements a high priority in their development cycle.
The Advanced Search option allows you to restrict your search by dates.
The Reader doesn’t write the vB software, nor the code for the SQL engine.
When I was writing multi-user software some years ago, I was often faced with the problem that a simple program that was ideal for a single-user system would be much less useful as more users signed on. The cure is to optimize for the situation. If vB or SQL locks the entire database during any search, and those searches can take more than just a second or two, a major change in the software is called for.
I see no hard & fast reason why the database has to be locked during a search. The simpliest way to write one is to lock, but there are other ways to accomplish the same task.
I recall a problem when I had to backup an entire company’s database but not affect the continuous, ongoing operation of the company. The solution was a routine that buffered all record changes and new additions until the backup task was done. Not a simple routine, but necessary under some circumstances.
Duh. But does it make the search op quicker? Depends on the organization and indexes, and that’s more system detail than I am privy to.
A good system, if indexed in date sequence, will avoid looking at records not in a specific date range. A poor system, or one not indexed on date, will have to look at ALL records and setting a date range will not affect the total search time.
Holy sugar puffs, Batman! You can’t be serious. That is a farcical way to implement a search algorithm. I don’t believe vBulletin locks up the database to perform a search, at least from what I have learned from the vBulletin Manual .
I think you are mistakened. Perhaps it appears to you that the database locks up because the servers become unresponsive when there are searches going. May I hypothesize that the default database index table has become too large for vBulletin to handle in a timely manner?
Either way, something like running Seekafile on the SDMB servers is the way to go. So, is the server running Windows 2003, Apache, or some version of Linux? I could try to whip up some kind of proof of concept modification if I know what we are working with.
Holy fuck, that’s probably me (or, at least, I was involved with the searches in the thread I presume you’re referring to), and boy am I embarrassed. I had no idea search was so taxing; it locks up the entire database and prevents posting? Seriously? That’s… I mean… Jesus…
Wow, I am never conducting another search again.
There’s something else causing the lockups. Some very large boards use vBulletin and don’t lock up when people search.
If so, then my professional WAG is that a tweak in the SQL implementation might be in order. Unfortunately, I do not have that much experience with SQL internals, but it would be worth looking into.
No time to reply in depth right now. All I’ll say for the moment is: When I first looked into this I too thought, jeez, this is stupid, there’s gotta be some simple fix. After lengthy inquiry I established that there isn’t. More later.
I would like more information on the checks that may or may not have been put in place to stop rogue spiders from accessing the SDMB. If these checks have not been implemented, they should be. There have been, and likely still are, websites which allow you to search this board. If it has gone to this extreme, then robots.txt is not enough. Punish violators of your TOS before you violate paying members.