The Anglo-Irish do speak with a near-English accent, but the “Irish” part of the identity is reflected in the fact that they aspirate. “Which” and “witch” are not homophones, and the ‘h’ is voiced in words like “hotel”. At least, this used to be the case, as noted by students of linguistics.
But the Anglo-Irish weren’t completely homogenous, of course, and the stereotype reflects speech and other mannerisms from the highest social ranks of Anglo-Irishry. Somebody like W B Yeats was Anglo-Irish, but we know from recordings that there were distinct traces of Sligo in his speech.