That’s a more-controlled situation. But it’s not a fully-controlled situation. Did you survey the girls during PE class or during regular class? Was it winter or summer? Rainy today or not?
Most of all, what question are you trying to answer? You can only understand the biases and hence the accuracy in detail of your survey methodology versus the specific question you’re trying to answer.
You don’t need to know much about the question to identify a crap survey with a crap methodology. But once we get past the crap surveys it gets harder to quantify the true metric of reliability of OK, good, or very good survey designs.
During the last election, some poll (Los Angeles newspaper?) consistently reported Trump as doing much better with blacks than any other poll, and consequently doing much better overall. Someone dug into this and found - the poll determined what minorities each respondent belonged to; they apparently randomly selected a respondent, young and black, who was extremely favorable to Trump. Since he was one of their only African-American respondents in the small sample in California, he skewed the entire poll to suggest Trump was doing really well there among black people. Then… they used the same people every week for their sample for weekly polls, so this guy consistently skewed the results instead of being a one-time blip.
It’s still important how those 50 were selected. In such a situation it’s at least conceivable you could generate a true random sample, because you could obtain a list of all the girls in the school and randomly select 50 from that list.
I believe that in a situation like that, where the sample is more than 5% of the population, the calculations involve a “finite population correction factor”; but the more statistically literate Dopers are probably better able to address that.
I assume they were mostly calling people from 9-5 M-F? To get people who are working you need to also call nights/weekends. Most surveys do very little calling 9-5
**@mdcastle **- the possible sources of bias I’d talk to the client about in this type of observational study would be:
[ul]
[li](As noted by Mr. Boink) where / when are the observations take place?[/li][li]locations around the gym class may be distorted since girls leaving gym class late in the day may wear their shorts home[/li]Where are the change rooms relative to the gym class? Do they get changed and walk through the school (where my observers are) to get to the track to excersize?
[li]First class in the morning - girls that have gym first class may wear shorts to school to save time changing.[/li][li]Lunch time - girls may wear shorts / jeans if they are leaving the school to go for lunch at the nearby mall. [/li][li]Is there a dance or some other formal activity (awards ceremony?) that may have girls dress up that day?[/li][li]Is there a “casual day” that day? My son normally wears a uniform but they get casual Fridays once a month - so the students wear jeans or shorts that day.[/li][/ul]
Also - **@ Banksiaman & Bijou Drain **- we often have that situation, our client will want a valid sample of all provinces in Canada of something like 26 to 28 year old females who completed university and are making +$60,000 / year. We often have the case that we complete places like Ontario and Quebec (lots of people) within a day, but smaller provinces like Newfoundland or PEI can take a couple weeks (simply because there are so few people). We flag that at the start of the project and let the client know we will devote X time and Y dollars and then stop, so the budget doesn’t get blown looking for the last 20 people we’d need to be to be valid. We’d either 1) press on 2) scrap the requirement or 3) relax it (non-uni. grades, less income, wider age group, depending on where we thought we were getting hung up.