I’m running some scores over 6 trials with each consecutive trial as its own matched-pair t-test i.e., the answers for test 1 and 2, test 2 and 3, test 3 and 4, test 4 and 5, and test 5 and 6.
I’m also running ANOVA to give a holistic view to the results and I’m also running Tukey as my post-hoc tests. But Tukey is showing that the significant scores from the t-tests are not significant.
I think that I have been very careful with the t-tests because I know when you run it on multiple trials you can easilly make a type I error, but by only looking at the scores from treatment to treatment e.g. the subject’s answers were significantly different between test 3 and 4, there should be some correlation between what the t-tests and Tukey tell me. What’s the difference and if the t-test show significance and Tukey doesn’t, what does that tell me?
Most post hoc tests try to keep the familywise chance of error below alpha. So if there are 6 comparisons, the FW alpha would be .05/6 = 0.008 per single comparison. This may be a bit of an oversimplification, and not all post hocs work the same. There are 15 Tukey comparisons in your example. I can’t remember, but Tukey might be one of those that checks largest first, e.g. if they are in order of means from 1-6, if 3 vs 6 is not significant, then 4 vs 5 will not be either, so it isn’t tested.
If there is a discrepancy between the two, that could indicate Type I error, although you can’t know this for certain, statistics being what they are. The Tukey is not significant because the critical value has been artificially increased to compensate for potential chance of error.
You can also do planned comparisons, which just lets you do the individual t-tests. You would normally only plan very few comparisons, and only ones of specific research interest.
Thanks, that’s what I thought. I think for the purposes of getting my dissertaion accepted, I will leave in the Tukey analysis because it does have a couple interesting results but mostly because I want the readers to know that I know about post-hoc analysis for ANOVA. I’m also going to keep the pairwise t-tests since they follow a chonological order and thus have some meaning within the context of the study.
Is it possible that part of the issue is that the matched-pair t-test is more specific than the more general ANOVA/Tucky therefore I would expect more significant a<0.05 with the t-tests?