For each rotation of Enigma it effectively generates one of the 26! (~4 x 10^26) possible overall substitution ciphers to encrypt one letter, correct? With a three wheel Engima without a reflector only 26^3 (17576) possible overall substitution ciphers are able to be generated out of the 26!.
With certain rotor wirings (say a straight passthrough A->A etc.) Engima will not generate a unique overall substitution cipher for each position/rotation. Whilst this is an extreme example my hunch is that not all rotor wirings/combination of rotor wirings will produce a different overall substitution cipher for each position/rotation. Whilst I guess most combinations will produce 17576 different overall substitution ciphers the others will only produce between 1 to 17576-1 different overall substitution ciphers?
In picking the various individual substitution ciphers used by each rotor (as seen in the link above) what analysis did the Germans perform to ensure no duplicate overall substitution ciphers for any two positions? Or what should they have done? For my example they could have possibly verified it by hand but for a four wheel engima there would have been simply too many permutations.
If there are 26! possible rotors how many combinations are not optimally strong for my three wheel, no reflector Enigma?
Part of the Enigma system was the day key, which specified that day’s permutation of rotors. (Initially in the army Enigma, the day key specified three rotors out of five available, for 60 available day keys.) A major element of cracking Enigma was cracking each day key, fast enough to be useful that day. Sometimes they succeeded.
As for how each rotor was wired, my guess is that the transposition was pure random.
I’m no math guru, so I can’t address the numbers. But I was pleased to discover John Keegan’s discussion of Enigma from his book Intelligence in War is available online. That may be of assistance to you.
In particular, somewhere in that book, he mentions that eventually someone realized that the internal wiring of Enigma was NOT actually increasing the number of possible permutations, thus simplifying the decryption by an order of something like 100 million (going by memory here).
From what I remember, the internal section of each rotor could be rotated relative to the outside. Thus, if A = J in the original position, rotating it one place turns it into B = K. This setting for each rotor was listed in the code books and changed every two days, I think.
Minor correction: Because of the way the design uses a reflector, an actual Enigma machine can never encrypt a letter as itself, and the reflector always has to do a substitution. So the premise is kind of flawed if we’re talking about a real Enigma, since you always need a reflector, and the total number of possible ciphers is much less. Can’t do the math quickly, but I’m guessing a few orders of magnitude less.
But I think this point is valid, whether we’re talking about a real Enigma or an idealized one without a reflector (that can encode a letter to itself). Some combinations of possible rotors will produce the same substitution cipher even though they are in different positions.
For example a rotor that swaps pairs of adjacent letters. After rotating one position, it gives a different substitution, but after the second turn, it’s back to the original substitution.
There are two interesting questions : first, is there a way to pick rotors so as to ensure that this doesn’t ever happen; and second if it did happen but rarely, could a code-breaker take advantage of this in any practical way (assuming that it’s rare enough not to make brute-forcing the combinations an option)?
You might be surprised. It’s actually 1.484 x 10^26 derangements of 26 items versus 4.033 x 10^26 permutations. So less, but not quite orders of magnitude less. However, I’m mostly sharing this fact to spread the knowledge that permutations without fixed points are called derangements.
To partially answer the OP. If c is the cyclic permutation (A -> B, B -> C, etc) and R is the permutation hardwired into your rotor. Choosing an R that commutes with any power of c (i.e. c^k R = R c^k) will reduce the period of your polyalphabetic cipher, which one would like to avoid.
This may or may not be a complete answer, but this certainly describes a subset of the “bad” rotors.
For my example they could have possibly verified it by hand but for a four wheel engima there would have been simply too many permutations.