I’m going to illustrate this problem using that noble Scottish beastie, the Haggis.

Now, as everyone knows, the Haggis in its natural state is a wide ranging and somewhat grumpy wee creature, with an interesting feature; in order to better navigate the bens and glens of it’s native home, a Haggis has shorter legs on one side than the other.

Let’s imagine that we start off with a population of 100 Haggis of which 100% are “right-handed”; that is, each and every Haggis has it’s longer legs on the right side of it’s body. However, Haggis have a stable genetic mutation which means that 10% of their kids will switch; they’ll be “left-handed”, with their left side legs being longer. Haggis (for currently unknown reasons) ensure themselves that the next generation will be the same overall size as the previous.

In the next generation of Haggis, we’ll end up with 90% right-handed, and 10% left-handed, 90 and 10 Haggis respectively. In the next generation, a further 10% of those 90 right-handed Haggis will switch, leaving us with 81 RHH and 9 LHH. But the switch gene also applies to the LHH, and from their population we end up with 9 LHH and 1 RHH, giving us overall 82 RHH and 18 LHH.

Let’s say we leave these Haggis around to breed for infinite generations. On average, what will be the ratio of RHH to LHH?

I think it will stabilize at 50% RHH and 50% LHH. Any population larger than 50 is going to shrink each generation until the influx from the other group’s mutations balances out its losses. That point of equilibrium is at 50-50.

Certainly once the population hits a 50/50 ratio, it will stay there – it’s a stable equilibrium point. the 50 RHs will produce 45 RH and 5 LH. The 50 LH will produce 45 LH and 5 RH – 50/50 again.

I strongly suspect (OK, I’m virutally certain) that there aren’t any other equilibrium points, and thus, as mks57 states, the population ratio will even out at 50/50, but I’m too pressed for time to actually do the math right now.

Let X[sub]n[/sub] be the fraction of right handed Haggises in the nth generation, therefore it has a value between 0 and 1. The formula to determine the fraction in the n+1th generation is:

For a stable population X[sub]n+1[/sub]=X[sub]n[/sub], so:

X[sub]n[/sub]=.5

The only stable population ratio is 50/50.

We take a look at the rate of change of the population ratio, which is X[sub]n+1[/sub]-X[sub]n[/sub].

X’=.1-.2*X[sub]n[/sub]

We see that the change is always positive when X[sub]n[/sub] is less than .5 and always negative when X[sub]n[/sub] is greater than .5. Thus, we have shown that the population always moves towards a .5 ratio and that .5 is the only steady ratio. We can even solve for the population as a function of n generations:

This looks like a classic example of a Markov chain, and the OP’s question is that of finding the steady state, which is a fairly standard problem. Markov chains are often modeled using matrices and matrix multiplication. If you’re familiar with matrix multiplication, you may find examples such as this or this enlightening.