It’s also folded into the name in Japanese. You can leave off the ~ken affix in casual speech, but most of the time it seems to be considered as part of the name. This also happens with transliterated names for other places and landmarks besides prefectures. You get things like signs reading “Nihonbashi Bridge.” The bashi part in Japanese means bridge, so the English, if literally translated, would mean Japan-bridge Bridge.
I’ve discussed this with people who are actually responsible for making some local signs, and apparently it’s common practice to leave the parts that mean bridge, road, mountain, etc. in the transliterated name. The thinking is that the name contains that bit, so it would sound odd to Japanese ears without it, and it aids communication when you’re dealing with people who don’t know each other’s languages well.
There’s also the common occurrence of having a village, town or city, and a prefecture with the same name. There’s a Saitama-shi (city) in Saitama-ken (prefecture). There’s a Saitama-mura (village) and for all I know there might also be a Saitama-machi (town) in Saitama-ken. Less confusing to never throw out the division bit unless both people can be sure of understanding what they’re referring to.