# Calling Zeldar, or any other statisticians: how many of us are there? **URL:** \ **Category:** About This Message Board\ **Created:** [March 16, 2010, 3:41pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661 "2010-03-16T15:41:37Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![aldiboronti](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@aldiboronti](https://boards.straightdope.com/u/aldiboronti)\ **Post date:** [March 16, 2010, 7:11pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/21 "2010-03-16T19:11:19Z")
I really don’t see how this could be super-secret information. Companies surely have no control over analysis of the public traffic on their websites. It’s not as if there’s any hacking involved. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 16, 2010, 7:14pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/22 "2010-03-16T19:14:07Z")
> [@NineToTheSky](#): > > Yes, but it’s not anyone’s usage that I’m interested in. Once someone has made a post in a given time frame, he’s on the scale. It’s the number of posters who have been active (and one post in this scenario is active) in a given period. Okay. Let’s assume you have the data mining app to do it. (I know so little about such things all I can do is pretend it could be made to work with what I do know about SDMB.) Using the User Profiles, you could extract every user whose latest activity was since some start date for your study. Count them. Save their names. Wait a span of time (day, week, month, whatever). Repeat the process. Compare the two data sets and either toss out the ones that aren’t common to the two or ascribe them to some second or third category to be monitored after a third collection that same length of time later. After that third collection you ought to have some feel for: 1. people who were active when you started but aren’t after the third cut 2. people were weren’t active back then that are now 3. people who have been active all along. A total of all the before-during-after sets ought to help in your decisions. I sense this is way too simplistic, but it’s about all I can contribute with my ignorance of data mining to work with. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 16, 2010, 7:15pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/23 "2010-03-16T19:15:42Z")
> [@aldiboronti](#): > > I really don’t see how this could be super-secret information. Companies surely have no control over analysis of the public traffic on their websites. It’s not as if there’s any hacking involved. I do agree. I was mostly being snide! 😃 ---
**Author:** ![hajario](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hajario/32/171_2.png) [@hajario](https://boards.straightdope.com/u/hajario)\ **Post date:** [March 16, 2010, 7:22pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/24 "2010-03-16T19:22:28Z")
> [@aldiboronti](#): > > I really don’t see how this could be super-secret information. Companies surely have no control over analysis of the public traffic on their websites. It’s not as if there’s any hacking involved. But if they want to make it against the rules, they can do it. I suppose that they couldn’t stop you from finding the information but they could ban you for disclosing it. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 16, 2010, 8:03pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/25 "2010-03-16T20:03:03Z")
**NineToTheSky** , I just had to go laugh myself to tears again reading [Anyone recall this classified ad from the Times?](http://boards.straightdope.com/sdmb/showthread.php?t=509561) which has to be in the Top Ten of all the fun threads I’ve been involved in. Thanks so much for that exercise in nonsense. ---
**Author:** ![RaftPeople](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RaftPeople](https://boards.straightdope.com/u/RaftPeople)\ **Post date:** [March 16, 2010, 8:25pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/26 "2010-03-16T20:25:35Z")
Just use automation software (I’ve used Automate5, not sure what the other ones are) to click every thread/page, save the page source as a text file. Then run a little script to process all of the text files (which contain all the posts) and dump each post into a database with user, datetime, etc. Then analyze your data. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 16, 2010, 8:30pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/27 "2010-03-16T20:30:54Z")
> [@RaftPeople](#): > > Just use automation software (I’ve used Automate5, not sure what the other ones are) to click every thread/page, save the page source as a text file. > > Then run a little script to process all of the text files (which contain all the posts) and dump each post into a database with user, datetime, etc. > > Then analyze your data. I just knew there had to be a simpler way! Where do you get **Automate5** these days? Does it come with a flanger? ---
**Author:** ![Ed\_Zotti](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@Ed\_Zotti](https://boards.straightdope.com/u/Ed_Zotti)\ **Post date:** [March 16, 2010, 8:33pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/28 "2010-03-16T20:33:47Z")
> [@NineToTheSky](#): > > Fair enough. But would you have any objections to us trying to work out that information? Nah, have at it. Just don’t do anything that’s going to crash the system, like having some bot submit 46,000 search queries (gawd, pls don’t do that). Things have been a little precarious lately, for reasons we’re still trying to ascertain. ---
**Author:** ![Giraffe](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/giraffe/32/129_2.png) [@Giraffe](https://boards.straightdope.com/u/Giraffe)\ **Post date:** [March 17, 2010, 1:20am UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/29 "2010-03-17T01:20:37Z")
> [@Ed\_Zotti](#): > > Nah, have at it. Just don’t do anything that’s going to crash the system, like having some bot submit 46,000 search queries (gawd, pls don’t do that). Things have been a little precarious lately, for reasons we’re still trying to ascertain. If you don’t object to people spidering the information, any reason not to simply enable the active posters option in vBulletin? It’s easy to do – just go to vBulletin Options / Forum Home Page Options and check the Show Total Number of Active Members box at the bottom. It would be nice to have a feel for the number of people currently participating in the community out of the 88,000 accounts that have been registered since the board started. ---
**Author:** ![NineToTheSky](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@NineToTheSky](https://boards.straightdope.com/u/NineToTheSky)\ **Post date:** [March 17, 2010, 9:05am UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/30 "2010-03-17T09:05:39Z")
> [@Zeldar](#): > > **NineToTheSky** , I just had to go laugh myself to tears again reading [Anyone recall this classified ad from the Times?](http://boards.straightdope.com/sdmb/showthread.php?t=509561) which has to be in the Top Ten of all the fun threads I’ve been involved in. > > Thanks so much for that exercise in nonsense. It’s my absolute pleasure. I have exciting news! **BigT** (may his name be forever blessed) has resurrected it again! There’s life in the old dog yet. ---
**Author:** ![NineToTheSky](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@NineToTheSky](https://boards.straightdope.com/u/NineToTheSky)\ **Post date:** [March 17, 2010, 9:14am UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/31 "2010-03-17T09:14:56Z")
> [@Ed\_Zotti](#): > > Nah, have at it. Just don’t do anything that’s going to crash the system, like having some bot submit 46,000 search queries (gawd, pls don’t do that). Things have been a little precarious lately, for reasons we’re still trying to ascertain. Actually, from this end, I’ve never seen the boards being more stable and responsive as they are now. But, at one search every five minutes, 46,000 searches would take 159.72 days. I’m up for it. Or, if we knew that there were 46,000 active posters, we could all do it at once. Only joking, Ed. **Giraffe** : > [@](#): > > _If you don’t object to people spidering the information, any reason not to simply enable the active posters option in vBulletin? It’s easy to do – just go to vBulletin Options / Forum Home Page Options and check the Show Total Number of Active Members box at the bottom. It would be nice to have a feel for the number of people currently participating in the community out of the 88,000 accounts that have been registered since the board started._ Independently of this thread, I’ve always wanted to see this. It would also help in determining the number of active posters (which may be why it’s not enabled). ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 17, 2010, 12:49pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/32 "2010-03-17T12:49:27Z")
To demonstrate my ignorance of bots and data mining in general, let me ask what’s wrong (both from SDMB’s viewpoint and as a technical task) with this approach. If you click [here](http://boards.straightdope.com/sdmb/member.php) you get **NineToTheSky** ’s profile page where, among other things, you see: Last Activity: Today 07:00 AM (or at least I did just now). Now if you were to write a script that started with [http://boards.straightdope.com/sdmb/member.php?u=1](http://boards.straightdope.com/sdmb/member.php) you’d get > [@](#): > > vBulletin Message > Invalid User specified. If you followed a valid link, please notify the administrator so you’d have to handle that condition. Okay. Increment the u= to 2 and you get > [@](#): > > Lynn Bodoni’s Profile but there’s no visible _Last Activity:_ which I would atttribute to the Administrator status. You’d have to handle that condition. Otherwise, just increment the u= number, check the Last Activity and when it hits a number/date greater than your desired Start Date, capture the stats for that user and store them. Keep going until the u= number of the latest member, which should be near the one on the opening page, which at this moment has these details: > [@](#): > > Threads: 533,609, Posts: 12,124,921, Members: 88,882 > Welcome to our newest member, daybeadacle and if you click on **daybeadacle** ’s name you get: “[http://boards.straightdope.com/sdmb/member.php?u=89212](http://boards.straightdope.com/sdmb/member.php)” which could serve for your Stop Point. (I won’t bother trying to guess why the 88,882 Members don’t match the u=89212 part of the URL, but I am mildly curious about that.) What am I missing here? No “search time” as such. No obvious strain on the hamsters. What? ---
**Author:** ![Lestrade](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@Lestrade](https://boards.straightdope.com/u/Lestrade)\ **Post date:** [March 17, 2010, 3:12pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/33 "2010-03-17T15:12:06Z")
> [@Zeldar](#): > > What am I missing here? No “search time” as such. No obvious strain on the hamsters. What? It would be very rude to make 88,882 page requests in a short time. To make that many politely, say 1 every 10 seconds, would take over 10 days of non-stop page loads. I think that that would be too fast, actually, I try to keep ~30 seconds between page requests without specific permission of the owners, so make that over a month. Bot usage looks very different from normal browser usage, and it can cause problems far sooner than you might expect. It would require far fewer requests to look at the number of posters that have posted in each thread that has been active in the last two days. It wouldn’t be much harder, programmatically, and it would require less bandwidth. I think the idea of iterating over each user id is a perfect example of something Ed would like us **not** to do. 🙂 ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 17, 2010, 3:23pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/34 "2010-03-17T15:23:21Z")
> [@Lestrade](#): > > It would require far fewer requests to look at the number of posters that have posted in each thread that has been active in the last two days. It wouldn’t be much harder, programmatically, and it would require less bandwidth. > > I think the idea of iterating over each user id is a perfect example of something Ed would like us **not** to do. 🙂 Better approach for sure. Glad I asked. Thanks. ---
**Author:** ![NineToTheSky](https://avatars.discourse-cdn.com/v4/letter/n/b5a626/32.png) [@NineToTheSky](https://boards.straightdope.com/u/NineToTheSky)\ **Post date:** [March 18, 2010, 11:41am UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/35 "2010-03-18T11:41:32Z")
Unfortunately, I have no programming skills whatsoever, so I can’t offer solutions to my OP. I’m worried that this ominous silence means that finding out how many active posters there are is not possible. Which is a shame. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 18, 2010, 12:03pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/36 "2010-03-18T12:03:11Z")
> [@NineToTheSky](#): > > Unfortunately, I have no programming skills whatsoever, so I can’t offer solutions to my OP. > > I’m worried that this ominous silence means that finding out how many active posters there are is not possible. Which is a shame. It’s another one of those “let’s tease these guys and just not reply to their thread” events. You braced for it? Let the nonsense begin, Who really cares how many there are? Lets’ say 2,000 and forget it. ---
**Author:** ![Zeldar](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zeldar](https://boards.straightdope.com/u/Zeldar)\ **Post date:** [March 18, 2010, 12:07pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/37 "2010-03-18T12:07:18Z")
Out of 36 posts, here’s the breakdown: ```auto Who Posted? Total Posts: 36 User Name Posts Zeldar 15 NineToTheSky 11 hajario 2 Ed Zotti 1 RaftPeople 1 Tom Tildrum 1 aldiboronti 1 Colibri 1 Giraffe 1 Lestrade 1 TubaDiva 1 Show Thread & Close Window ``` ---
**Author:** ![xash](https://avatars.discourse-cdn.com/v4/letter/x/c6cbf5/32.png) [@xash](https://boards.straightdope.com/u/xash)\ **Post date:** [March 18, 2010, 12:28pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/38 "2010-03-18T12:28:06Z")
> [@](#): > > (I won’t bother trying to guess why the 88,882 Members don’t match the u=89212 part of the URL, but I am mildly curious about that.) My guess is that users were deleted from the system at one point or another, or for one reason or another. User IDs are not re-used, so there are more User IDs than there are existing users in the system. For example, [http://boards.straightdope.com/sdmb/member.php?u=1](http://boards.straightdope.com/sdmb/member.php) does not exist. ---
**Author:** ![BigT](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bigt/32/12044_2.png) [@BigT](https://boards.straightdope.com/u/BigT)\ **Post date:** [March 18, 2010, 12:33pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/39 "2010-03-18T12:33:42Z")
> [@xash](#): > > My guess is that users were deleted from the system at one point or another, or for one reason or another. User IDs are not re-used, so there are more User IDs than there are existing users in the system. > > For example, [http://boards.straightdope.com/sdmb/member.php?u=1](http://boards.straightdope.com/sdmb/member.php) does not exist. Yeah. I actually went through the first few trying to find **Cecil Adams** so I could do a search on his posts to compare him with a certain other member. Then it dawned on me that I could just use the advanced search. My experience with u=1 is that that is often some sort of administrative account that gets deleted once the board is up and running. ---
**Author:** ![Ed\_Zotti](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@Ed\_Zotti](https://boards.straightdope.com/u/Ed_Zotti)\ **Post date:** [March 18, 2010, 12:41pm UTC](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661/40 "2010-03-18T12:41:53Z")
> [@Giraffe](#): > > If you don’t object to people spidering the information, any reason not to simply enable the active posters option in vBulletin? It’s easy to do – just go to vBulletin Options / Forum Home Page Options and check the Show Total Number of Active Members box at the bottom. It would be nice to have a feel for the number of people currently participating in the community out of the 88,000 accounts that have been registered since the board started. I tried this, to see what would happen. I set the time horizon to 60 days. What I saw was: “Active members: 0.” Obviously there’s some subtlety I’m overlooking. I also know that when I did a “top statistics” search, to see how THAT worked, the board was out of action for a good five minutes. Coincidence? Maybe. All I know is, I ain’t messing with this thing. [Previous page](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661.md?page=1) [Next page](https://boards.straightdope.com/t/calling-zeldar-or-any-other-statisticians-how-many-of-us-are-there/532661.md?page=3)