I believe this is generally hashed out and dealt with in terms of Evolutionary Game Theory but I couldn’t find the specific example or issue that I was thinking of being discussed.
Supposing you have a type of prey animal whose response to a predator attack is some sort of group response, e.g. mobbing, as in birds or buffalo. On the whole, once that has been established as the group strategy, the group is presumably better off than had they not established this behaviour, and once the group is better off then each individual is also better off.
But the mutation which causes this behaviour would not have occurred simultaneously in all members of the group at the same time. And if one prey animal decides to confront the predator by itself while the rest flee, then that animal has a disadvantage as compared to the rest. So suppose one prey animal had some mutation or other genetic change which caused it to have a more aggressive approach to predators than the group, with the rest of the group being still more timid, then that animal is less likely to survive than the rest, and this gene is less likely to be passed on than that of the more passive or timid members.
So the question is basically: how does the ball get rolling?
Once you have two different populations, in one of which the altruistic approach has become the norm and in one of which it has not, you can easily see how the former would gradually displace the latter. But the question is how the altruistic approach can ever transfer from an individual to a population.
Perhaps it started with kin altruism- a group of brothers and sisters mob together to defend themselves from predators after inheriting a random mutation from their parents. By defending close genetic kin, they’re protecting most of their own genes (as I learned from Dawkins’ “The Selfish Gene”). A higher portion of this family (and thus the genes) survives to reproduce, and at some point perhaps there’s another small mutation that extends the altruism from kin to herd/flock.
One possibility is a recessive gene. Any spread through the population at first would be purely random because it can’t be expressed until two carriers breed together, and even then will only be expressed in 1/4 of their offspring.
You’re also assuming that it’s a single gene, and that the trait either exists, or it doesn’t. Maybe it started with an inclination to help the group only when it came at little or no personal cost, which then gradually strengthened until it turned into the whole “needs of the many” thing.
I’d assume that the first trait developed was a tendency to stay in the herd, so that there are other targets available. Once developed, it would be the strongest urge. From there, different individuals could have a different tendency to stand ground or mob, perhaps developed as Chronos suggested. If you have a small herd and one individual with a strong tendency, plus a couple of relatives with a weaker tendency, and then a few more who wouldn’t mob unless most of the herd was doing it first - the few left who have no urge at all might not want to leave the herd and also might be better targets if they broke and ran.
How dangerous is mobbing? That is, in animals who mob, do most do so cautiously?
There’s also the question of whether the premise is true: does one individual ‘mobbing’ really face a disadvantage? Maybe even one individual flying straight at a predator confuses the predator enough that it’s advantageous for that individual.
And there are lots of group prey behaviors that it’s pretty clear wouldn’t be detrimental even if only one individual did it. For instance, if predators are much less effective without surprise, letting the predator know they’ve been spotted and it’s not worth attacking is beneficial to a single individual even if no other member of the group does it [the predator has to pick up on the signal, but that’s another evolutionary discussion].
And, I’m not sure what ‘mobbing’ means for either bison or true buffalo, but I imagine in both cases, the better strategy for an adult is to stand with their pointy end towards an attacker rather than running away, whether in a group or all alone.
I would imagine it is similar to flocks of starlings. Each individual is only following local rules, and it is easy to see how that could come about, as in protecting a calf, or perhaps some kind of food seeking. Anyway, when enough can do it, it begins to look like an organized group, where actually each member is following rules concerning others close to them.
Any time you have a very rare mutation, it’s going to be more subject to genetic drift (ie random chance) than positive selection until it spreads to an appreciable proportion of the population. In a situation like this, I’d expect that the rare mutation would spread by pure chance for a few generations - perhaps, by sheer luck, the carriers don’t get attacked very much, or manage to survive their seemingly suicidal attack. Perhaps it IS suicidal and the bird dies, but he’s already managed to reproduce. That sort of thing. In the cases you describe, though, I don’t think it would take a huge mob to do the job. A set of siblings, all carrying the mutation, could be enough to get it done. That would mean it would only need to survive a generation or two before it could confer an advantage. There are plenty of deleterious mutations that survive that long in populations.
A lot of traits started out doing one thing and then were co-opted for something else. For example, no one truly understands how flight started in insects, but there is speculation that wings grew for thermal regulation, gradually got adopted for gliding and then acquired the ability to flap. Anyone who has seen squirrels jump from tree to tree could easily imagine a progression: squirrels to gliding squirrels to bats.
Anyway, a cooperative in-group behavior could obviously be modified to become mobbing of predators. Of course, this is all conjecture; we have no real evidence.
Also, a lot of altruistic behaviour is not suicidal, just risky. And many times not even particularly risky. Many predators go for the weakest in the herd, not the one that is so bold it is attacking the predator.
Actually, it’s believed that external gill structures (still used by some insect nymphs for underwater respiration) evolved into wings that water-dwelling insects used for locomotion while skimming on the water surface (like a bayou fanboat). Even quite small winglets are effective for such a purpose. From there, with the wing musculature and motion well developed, it’s a smaller step from high-speed skimming to flight.
In horses, this ‘more aggressive approach’ is also reflected in mating behavior within the herd. So the aggressive stallion mates with more of the mares in the herd, and more often – so it’s more likely that his genes will be passed on. Presumably, the evolutionary advantage of this outweighs the increased risk from an aggressive response toward predators.
This was my thought. The wolf or cheetah is going to go for the one running away, too busy to aim a kick or gore with horns. The one facing down the predator could cause problems during the confrontation. Plus, animals like canines have instinctive “chase the fleeing prey” reflexes, so one or a few who stand and confront will be left behind while the predator chases the fleeing obvious dinner.
Plus, as T-Bon points out, it could be a mutation of mating dominance instinct. The crazy aggressive behaviour that works well for keeping rivals away from your female(s) also works for keeping others away from your flank steak.
And, it could have worked its way up from a strategy to deal with smaller pests. While it’s risky to confront a pack of large wolves solo, confronting a smaller predator that takes a shine to veal jr. probably is an effective survival strategy. In fact aggression is a common tactic for many animals against less risky pests - heck, I’ve been chased by geese and ducks even.