The evolution of cooperation is a fundamental problem in biology, especially for non-relatives, where indirect fitness benefits cannot counter within-group inequalities. Multilevel selection models show how cooperation can evolve if it generates a group-level advantage, even when cooperators are disadvantaged within their group. This allows the possibility of group selection, but few examples have been described in nature. Here we show that group selection can explain the evolution of cooperative nest founding in the harvester ant Pogonomyrmex californicus. Through most of this species’ range, colonies are founded by single queens, but in some populations nests are instead founded by cooperative groups of unrelated queens. In mixed groups of cooperative and single-founding queens, we found that aggressive individuals had a survival advantage within their nest, but foundress groups with such non-cooperators died out more often than those with only cooperative members. An agent-based model shows that the between-group advantage of the cooperative phenotype drives it to fixation, despite its within-group disadvantage, but only when population density is high enough to make between-group competition intense. Field data show higher nest density in a population where cooperative founding is common, consistent with greater density driving the evolution of cooperative foundation through group selection.

Human societies are unique in the level of cooperation among non-kin. Evolutionary models explaining this behavior typically assume pure strategies of cooperation and defection. Behavioral experiments, however, demonstrate that humans are typically conditional co-operators who have other-regarding preferences. Building on existing models on the evolution of cooperation and costly punishment, we use a utilitarian formulation of agent decision making to explore conditions that support the emergence of cooperative behavior. Our results indicate that cooperation levels are significantly lower for larger groups in contrast to the original pure strategy model. Here, defection behavior not only diminishes the public good, but also affects the expectations of group members leading conditional co-operators to change their strategies. Hence defection has a more damaging effect when decisions are based on expectations and not only pure strategies.