I’m reluctant to get into this subject because it typically draws trolls and haters who know just enough science to sound impressive without actually saying anything one way or another. I’ve spent just over a hundred hours working on reviewing this subject, and here are some references to consider. Of the ones listed, Emory, Haraldsen, Ludens, Hengstschläger, and Savic probably contain the most anti-physical or anti-genetic evidence. The rest contain pro-physical or pro-genetic evidence.
Balen, Adam H., et al. “Polycystic ovaries are a common finding in untreated female to male transsexuals.” Clinical Endocrinology 38.3 (1993): 325-329.
Bao, Ai-Min and Swaab, Dick F. “Sexual differentiation of the human brain: Relation to gender identity, sexual orientation and neuropsychiatric disorders” Frontiers in Neuroendocrinology 32 (2011): 214–226.
Berglund, H. et al. “Male-to-Female Transsexuals Show Sex-Atypical Hypothalamus Activation When Smelling Odorous Steroids” Cerebral Cortex 18 (August, 2008): 1900—1908.
Cantor, James M. “New MRI Studies Support the Blanchard Typology of Male-to-Female Transsexualism” Arch Sex Behav 40 (2011): 863–864.
Coolidge, Frederick L., Linda L. Thede, and Susan E. Young. “The heritability of gender identity disorder in a child and adolescent twin sample.” Behavior Genetics 32.4 (2002): 251-257.
Dessens, Arianne B., et al. “Prenatal exposure to anticonvulsants and psychosexual development.” Archives of Sexual Behavior 28.1 (1999): 31-44.
Dorner, Gunter, et al. “Genetic and Epigenetic Effects on Sexual Brain Organization Mediated by Sex Hormones.” Neuroendocrinology Letters 22.6 (2001): 403-409.
Emory, Lee E., et al. “Anatomic variation of the corpus callosum in persons with gender dysphoria.” Archives of Sexual Behavior 20.4 (1991): 409-417.
Fontanari, Anna-Martha V. et al. “Serum concentrations of brain-derived neurotrophic factor in patients with gender identity disorder” Journal of Psychiatric Research (2013): 1-3 (advance copy).
Futterweit, Walter, Richard A. Weiss, and Richard M. Fagerstrom. “Endocrine evaluation of forty female-to-male transsexuals: Increased frequency of polycystic ovarian disease in female transsexualism.” Archives of Sexual Behavior 15.1 (1986): 69-78.
Gooren, Louis “The biology of human psychosexual differentiation” Hormones and Behavior 50 (2006): 589–601.
Green, Richard, and Eric B. Keverne. “The disparate maternal aunt–uncle ratio in male transsexuals: an explanation invoking genomic imprinting.” Journal of Theoretical Biology 202.1 (2000): 55-63.
Green, Richard. “Family cooccurrence of “gender dysphoria”: Ten sibling or parent–child pairs.” Archives of Sexual Behavior 29.5 (2000): 499-507.
Haraldsen, I. R. et al. “Sex-sensitive cognitive performance in untreated patients with early onset gender identity disorder” Psychoneuroendocrinology 28 (2003): 906–915.
Hare, Lauren et al. “Androgen Receptor Repeat Length Polymorphism Associated with Male-to-Female Transsexualism” Biol Psychiatry 65.1 (January 1, 2009): 93–96.
Hengstschläger, Markus, et al. “Sex chromosome aberrations and transsexualism.” Fertility and sterility 79.3 (2003): 639-640.
Hines, Melissa, Charles Brook, and Gerard S. Conway. “Androgen and psychosexual development: Core gender identity, sexual orientation, and recalled childhood gender role behavior in women and men with congenital adrenal hyperplasia (CAH).” Journal of Sex Research 41.1 (2004): 75-81.
Italiano, M. “Comment on Cantor” Arch Sex Behav 41 (2012):1079.
Khandelwal, Ashish et al. “A 47,XXY Female with Gender Identity Disorder” Arch Sex Behav 39 (2010):1021–1023.
Kruijver, Frank P. M. et al. “Male-to-Female Transsexuals Have Female Neuron Numbers in a Limbic Nucleus” The Journal of Clinical Endocrinology & Metabolism 85.5 (2000): 2034-2041.
Luders, Eileen et al. “Regional gray matter variation in male-to-female transsexualism” Neuroimage 46.4 (July 15, 2009): 904–907.
Meyer-Bahlburg, Heino FL. “Transsexualism (“Gender Identity Disorder”)–A CNS-Limited Form of Intersexuality?.” Hormonal and Genetic Basis of Sexual Differentiation Disorders and Hot Topics in Endocrinology: Proceedings of the 2nd World Conference. Springer New York, 2011.
Oh, Seok-Kyun et al. “Brain Activation in Response to Visually Evoked Sexual Arousal in Male-to-Female Transsexuals: 3.0 Tesla Functional Magnetic Resonance Imaging” Korean J Radiol 2012;13(3):257-264
Rametti, Giuseppina et al. “White matter microstructure in female to male transsexuals before cross-sex hormonal treatment. A diffusion tensor imaging study” Journal of Psychiatric Research 45 (2011): 199-204.
Rametti, Giuseppina et al. “Effects of androgenization on the white matter microstructure of female-to-male transsexuals. A diffusion tensor imaging study” Psychoneuroendocrinology 37 (2012): 1261—1269.
Sabalis, Robert F. et al. “The Three Sisters: Transsexual Male Siblings” Am J Psychiatry 131 (August 8, 1974): 907-909.
Sadeghi, Majid, and Ali Fakhrai. “Transsexualism in female monozygotic twins: A case report.” Australasian Psychiatry 34.5 (2000): 862-864.
Savic, Ivanka and Arver, Stefan. “Sex Dimorphism of the Brain in Male-to-Female Transsexuals” Cerebral Cortex 21 (November, 2011): 2525—2533.
Swaab, D.F. “Sexual differentiation of the human brain: relevance for gender identity, transsexualism and sexual orientation” Gynecol Endocrinol 19 (2004): 301–312.
Yokota, Y. et al. “Callosal Shapes at the Midsagittal Plane: MRI Differences of Normal Males, Normal Females, and GID” Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference Shanghai, China, September 1-4, 2005.
Zhou, Jiang-Ning, et al. “A Sex Difference in the Human Brain and its Relation to Transsexuality.” Nature 378.6552 (1995): 68-70.
Zubiaurre-Elorza, Leire et al. “Cortical Thickness in Untreated Transsexuals” Cerebral Cortex Advance Access, published August 31, 2012.
Zucker, Kenneth J., et al. “Psychosexual development of women with congenital adrenal hyperplasia.” Hormones and Behavior 30.4 (1996): 300-318.