Back in http://www.straightdope.com/columns/030418.html, the Perfect Master wrote
After I whined a bit, he addressed me (yay!) and said
and then modified the article to read
Ok, that is technically accurate, in the sense that Dean did say that. But an analysis of Dean’s study finds several serious flaws that I’ll lay out below. I propose that this is not a good enough study to draw the conclusion drawn.
First off, let me define what I believe is in contention. I propose simply that elements of personality are reflected in handwriting. That’s all. And that’s exactly what is claimed is not true in the column (“it was worthless as a predictor of personality”). I specifically do not claim
[ul]
[li]that this is useful for any purpose, such as psychiatric evaluation or personnel selection or evaluation.[/li][li]that people who practice handwriting analysis are either honest, scientific or consistent in their evaluation[/li][li]that ALL or even most personality traits that are claimed to be reflected by some are actually reflected.[/li][/ul]
All right, let’s dig in.
In the introduction, Dean presents quotes from the pro-Graphology side, and the “Scientific” side. Seeing as how the quotes on the pro-side include quotes from psychics and contain claims like “Handwriting analysis can help you in your search for whatever it is you want out of life”, and seeing how Dean labels the opposing side as “Views of Scientists,” Dean sets the stage that he’s about to debunk some extremists making extreme claims, and from that result move on to claim the entire field is fraudulent.
Next, Dean lays out what he will test. He is clear that he will not test “Is graphology true? Does it work? Is it real?,” but instead test “about extent.” Which is a fair thing to test, but that isn’t what is claimed in Cecil’s column. Cecil’s column claims that the answers to “Is it true? Does it work? Is it real?” are no, no and no. Which should invalidate this study right off the bat. But there’s a lot more.
Next, Dean throws Reliability into the mix. In other words, Dean evaluates how similar two Handwriting Analysts evaluations are of a single sample (or of how one Analyst compares against himself in a later evaluation). Which is also an interesting test, but again not what is in dispute here. Just because two graphologists come to different conclusions does not in any way refute the idea that one of the conclusions was accurate and reproducible.
Now comes Dean’s boldest act of non-science in this study. First, he states
Wow! Those three paragraphs basically seal the claim that some personality is reflected in handwriting, with an especially strong effect. And not just any personality traits; the two “biggest determinants of human behavior and destiny” in his words. As I say, that should seal it, so how does Dean deal with this obstacle?
He removes these factors from his study
That’s right. The two most important personality traits by his own definition, which have a very strong effect are removed from the study, leaving traits with weaker (or negative) effects in the remaining averages. Why? Dean doesn’t say. He just casually mentions that they were excluded and marches forward.
Now for the second boldest act of non-science: Dean does not define what personality traits are accepted as part of his analysis. (except the two most important by his own definition, which were discarded without reason). None. Are some of the studies predicting what a reasonable studier of the field would consider to have strong effect sizes such as extraversion? Or ones that a reasonable studier would consider having no effect (or even negative effect) such as whether the subject as an employee will sell his quota next quarter. Oh wait, that particular one is listed as in the study in a side-note, but I digress.
Using this data, of which we know nothing about except that it has dis-included the most important and most favorable data, Dean concludes that Handwriting has a low effect size in predicting whatever it is he’s testing it to predict. Which he describes as “undeniably dismal” and goes on to quote the typical arguments against his conclusion, but of course don’t mention the “what are you actually studying?” one.
Dean then goes on to demonstrate that Handwriting Analysis can predict IQ with a .29 effect size (which is very good, slightly below the effect of comparing how a score on a law school ability text will predict a first year law grade). However, he uses this to demonstrate a negative point, that this prediction of IQ is nowhere near as accurate as actually taking an IQ test for example. Which is true, but not relevant towards our inquiry. In fact, quite the opposite; we are looking for a correlation, and Dean has found it. The fact that there are other better ways of determining personality is not under question.
Finally, Dean concludes that yes, graphology is valid, but the validity factor is not very high (again because he’s doctored and hidden the input), and the reliability is low (which as discussed is not relevant to the discussion of whether it’s a predictor), and ultimately he says it is “not useful” because “other methods are better.” Which is true (it doesn’t stack up against an interview with a psychiatrist for example), but irrelevant to the point that yes, handwriting analysis does predict personality.
By the way, one of the first things one notices about Dean’s text is that better then half of it isn’t about handwriting analysis, but rather about the math behind statistics. Which is not bad in itself. But it brings back an immediate image of The Bell Curve, which took a similar tack (and was also severely attacked on the question of credibility). When one tells you over and over that this is scientific, you start to wonder just how scientific it really is.