Detecting differential viability selection between environments by analysis of compositional differentiation at different levels of genetic integration

Viability selection can be detected directly in an environment when the genotypes of the individuals at one ontogenetic stage (e.g. seeds) and the genotypes of the survivors at a later stage are both known, but genotypes at the earlier stage often cannot be determined. In this case, differential viability selection between environments can be detected as differences in the distributions of genetic types among survivors growing in different environments, provided that the survivors stem from random samples of seeds from the same base population (e.g. seed lot). Since common F_ST-outlier methods for detecting selected gene loci use only allele frequencies, selection that affects the higher hierarchical levels of genetic integration (single- or multi-locus genotypes) without changing allele frequencies is not noticed. A new method for detecting differential viability selection at any level of genetic integration enables discovery of elementary mechanisms of selection that older methods miss. It is based on two measures of compositional differentiation between environments. δ_SD measures qualita-tive differences between distributions of genetic types at any given integration level without regarding differences in their constituent alleles, while Δ_SD measures quantitative differences between the same distributions by additionally considering the genic differences. The difference between these measures expresses the degree to which the patterns of gene association in the genotypes differ between environments. The P-values of all measures are estimated by permutation analysis under the assumption that survivors were randomly assigned to environments. Significance indicates the occurrence of differential viability selection at the loci. As a case study, a field study of viability in juvenile beech (Fagus sylvatica L.) for twelve enzyme loci is reanalyzed. It turns out that the significant differential selection for genotypes detected at three loci can be attributed to three combinations of selective effects: on alleles only (SKDH-A), mostly alleles but also association patterns (LAP-A); interaction of effects on alleles and association patterns that are non-significant when viewed separately (AAT-B).