Why Transform Resistance Surfaces?

When modeling landscape connectivity, we start with a raw environmental layer (perhaps canopy cover, elevation, percent impervious surface, or NDVI) and convert it into a resistance surface. The simplest approach would be to use the raw values directly: higher canopy cover means lower resistance for a forest-dependent species, so we could just invert the raw values. But this assumes a linear relationship between the raw covariate and the movement cost it generates, and linear relationships are rarely the right biological assumption.

Consider a forest-dependent salamander moving across a landscape of varying canopy cover. At low canopy cover (say, 10–30%), even small increases in canopy might dramatically reduce resistance: the difference between an open field and a sparse woodlot matters enormously. At high canopy cover (80–100%), the additional canopy may have little further effect; all dense forest provides similarly good habitat, and the marginal benefit of an extra 5% canopy is small. This pattern (steeply decreasing resistance at low values, leveling off at high values) is a Monomolecular relationship.

Alternatively, consider a road-crossing bird. At very low road density, roads are rare enough that they impose little resistance. At moderate road density, roads fragment the landscape and substantially impede gene flow. At very high road density (dense urban areas), the bird simply doesn’t attempt to cross; resistance is high and roughly constant. This peaked relationship (low resistance at low and high values, high resistance at intermediate values) is a Ricker pattern.

The eight transformation families in ResistanceGA2 provide a flexible vocabulary for these ecological relationships. Rather than choosing the transformation a priori, the genetic algorithm searches the parameter space and identifies which shape and parameter values best explain the observed genetic differentiation.

The Transformation Parameter Space

Every continuous-surface transformation in ResistanceGA2 is defined by two parameters:

The Eight Transformation Families

The eight families are organized into two groups: Monomolecular and Ricker, each with four variants produced by inverting (flipping the y-axis) and reversing (flipping the x-axis) the base curve. The variants allow the GA to represent all four possible “directions” of monotone and peaked responses.

We illustrate each family using the raster_orig continuous surface with illustrative parameters (shape = 2.5, max = 100).

Monomolecular Families

Monomolecular

Mathematical form: resistance = max * (1 - exp(-x / shape)) + 1

The Monomolecular curve rises steeply from 1 at low values of the covariate, then decelerates and asymptotes near max + 1. It is the most commonly appropriate transformation for resistance modeling because many ecological variables have diminishing-returns effects.

Biological interpretation: Resistance increases with the covariate and saturates at high values. Suitable when the landscape variable represents a cost that increases with its value but has an upper limit. Examples include:

• Percent impervious surface: small amounts cause some resistance, but above ~80% impervious, additional pavement adds little marginal cost (the landscape is already effectively impassable for most terrestrial species).
• Temperature deviation from optimum: small deviations are tolerable; large deviations are uniformly unfavorable.

When to use: When you expect resistance to increase with the covariate and saturate at high values. This is the default and most ecologically conservative choice.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Monomolecular",
  marginal.plot  = FALSE
)

Monomolecular transformation: resistance increases with covariate value and saturates at high values.

---

Reverse Monomolecular

Mathematical form: Same as Monomolecular, but applied to the reversed covariate values (i.e., the curve is flipped left-to-right).

Biological interpretation: Resistance decreases with the covariate and saturates at low values. Suitable when the landscape variable represents a habitat quality that increases movement ease. Examples include:

• Forest cover: dense forest is easy to traverse for forest specialists; the effect saturates. Once you have 80% canopy, additional canopy adds little extra benefit.
• NDVI (greenness): high NDVI = suitable habitat = low resistance for many terrestrial species.

When to use: When you expect resistance to decrease with the covariate and saturate at high values of habitat quality.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Reverse Monomolecular",
  marginal.plot  = FALSE
)

Reverse Monomolecular: resistance decreases with covariate value and saturates at high values.

---

Inverse Monomolecular

Biological interpretation: Resistance decreases with the covariate but does so rapidly at low values and then levels off at high values. Similar to Reverse Monomolecular, but the approach is from a high-resistance baseline rather than a high-resistance peak. Suitable when:

• Movement ability increases sharply at first with the covariate (e.g., a threshold amount of moisture is needed for dispersal), then levels off.

When to use: When a small increase in covariate value rapidly reduces resistance, but additional increases have little further effect.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Inverse Monomolecular",
  marginal.plot  = FALSE
)

Inverse Monomolecular: resistance decreases rapidly at low covariate values, then stabilizes.

---

Inverse-Reverse Monomolecular

Biological interpretation: Resistance increases with the covariate, doing so rapidly at high values and leveling off at low values. This is the mirror image of the Inverse Monomolecular. Suitable when:

• A threshold must be crossed before resistance rises substantially (e.g., a road network only becomes a barrier once road density exceeds a critical level).

When to use: When resistance increases sharply only at high covariate values, with a near-flat response at low values.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Inverse-Reverse Monomolecular",
  marginal.plot  = FALSE
)

Inverse-Reverse Monomolecular: resistance remains low at low covariate values and rises sharply at high values.

---

Ricker Families

The Ricker family produces peaked (unimodal) curves. For standard Ricker, resistance is highest at intermediate values of the covariate and lowest at the extremes; the inverse variants produce the opposite pattern (lowest at intermediate, highest at extremes). They were adapted from the Ricker stock-recruitment model in fisheries science.

Caution: Ricker transformations are more flexible than Monomolecular families and can fit a wider variety of curve shapes, including near-monotone shapes when shape parameters are extreme. This flexibility comes at the cost of increased overfitting risk. Use Ricker families only when you have a biological hypothesis that predicts a peaked resistance response.

Ricker

Mathematical form: resistance = max * x * exp(-x / shape) + 1

This creates a bell-shaped curve with a peak at intermediate covariate values. The peak location is at x = shape (larger shape values shift the peak toward higher covariate values).

Biological interpretation: Resistance is highest at intermediate covariate values, making those areas difficult to traverse. Movement is easier at the low and high extremes of the covariate range. Examples include:

• Habitat fragmentation: A species might move readily through continuous forest (low fragmentation) or through open agricultural areas (high fragmentation), but struggles in the mixed suburban-forest transition zone (intermediate fragmentation) with scattered trees and dense development.
• Elevation: A generalist species might move effectively at both low elevations and high peaks, but face high resistance at mid-elevations if that zone represents an ecotonal transition with poor habitat quality.
• Vegetation heterogeneity: A species preferring uniform patches might experience high resistance in areas of intermediate patchiness, where neither homogeneous habitat nor fully structured mosaics provide suitable movement corridors.

When to use: When you have a clear biological hypothesis predicting that intermediate covariate values represent the most difficult-to-traverse landscape, not simply because the Ricker curve provides a better statistical fit.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Ricker",
  marginal.plot  = FALSE
)

Ricker transformation: resistance peaks at intermediate covariate values.

---

Reverse Ricker

Biological interpretation: Like Ricker, this produces a peaked resistance curve where intermediate values have highest resistance. The difference is that the peak is shifted toward higher covariate values compared to Ricker. This represents a scenario where intermediate-to-high covariate values are most difficult to traverse.

Examples include:

• Canopy cover: A species might move readily through open areas and moderately through partial canopy, but struggle in the dense-to-very-dense forest zone (intermediate-high cover) if that represents impenetrably thick understory.
• Disturbance history: Resistance highest in recently disturbed areas (intermediate time since disturbance) while being lower in very old-growth and newly cleared patches.

When to use: When biological reasoning predicts a peaked resistance response centered toward the higher end of the covariate range, rather than the lower-intermediate peak produced by Ricker.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Reverse Ricker",
  marginal.plot  = FALSE
)

Reverse Ricker: resistance is lowest at intermediate covariate values.

---

Inverse Ricker

Biological interpretation: The inverse of Ricker, producing a trough (U-shaped curve) where resistance is lowest at intermediate covariate values and highest at the extremes. This represents scenarios where intermediate conditions facilitate movement while extreme conditions resist it.

Examples include:

• Elevation: A generalist species might move easily through mid-elevation areas but face resistance at both low (swampy/waterlogged) and high (sparse alpine) extremes.
• Disturbance: A pioneer species might move readily through intermediate disturbance (early successional patches with favorable structure) but struggle in undisturbed forest and heavily degraded areas.

When to use: When your biological hypothesis predicts that intermediate covariate values represent the easiest-to-traverse landscape, with resistance increasing toward the extremes.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Inverse Ricker",
  marginal.plot  = FALSE
)

Inverse Ricker: peaked resistance response with inverted scaling.

---

Inverse-Reverse Ricker

Biological interpretation: Like Inverse Ricker, this produces a U-shaped trough where intermediate covariate values are easiest to traverse. The difference is that the trough is shifted toward higher covariate values, representing low resistance in intermediate-to-high regions.

Example: A forest specialist that moves readily through moderate-to-high canopy cover but struggles in open areas and in extremely dense, impenetrable forest.

When to use: When biological reasoning predicts a troughed (U-shaped) resistance curve centered toward the higher end of the covariate range. This is one of the rarer ecological scenarios and should only be used with a clear hypothesis.

Plot.trans(
  PARM           = c(2.5, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Inverse-Reverse Ricker",
  marginal.plot  = FALSE
)

Inverse-Reverse Ricker: complex peaked response with inverted and reversed scaling.

---

Comparing All Eight Families

The figure below shows all eight transformation families side by side using the same illustrative parameters (shape = 2.5, max = 100) applied to the range of the cont_orig surface. The purpose is to show the geometric diversity of the available curves, not to suggest these parameters are optimal for any real analysis.

ggplot(curve_df, aes(x = original, y = transformed)) +
  geom_line(linewidth = 0.9) +
  facet_wrap(~ Transformation, ncol = 2, scales = "free_y") +
  theme_bw() +
  labs(
    x = "Original covariate value",
    y = "Transformed resistance"
  ) +
  theme(
    strip.text  = element_text(face = "bold", size = 8),
    axis.text   = element_text(size = 8),
    axis.title  = element_text(size = 10)
  )

All eight transformation families at shape=2.5, max=100. Top row: Monomolecular family. Bottom row: Ricker family. Within each row: base, reverse, inverse, and inverse-reverse variants.

Key observations from this comparison:

1. Monomolecular families are all monotone. They produce one-directional responses: resistance either increases or decreases with the covariate, but never both.
2. Ricker families produce peaks or troughs. They can represent non-monotone responses where intermediate covariate values have a qualitatively different resistance than the extremes.
3. Inverse and Inverse-Reverse variants rescale the y-axis to ensure resistances are always positive and in the [1, max+1] range. The shapes are ecologically distinct from their non-inverse counterparts despite superficial visual similarity at some parameter values.

The Flat Distance Transformation

In addition to the eight transformation families, ResistanceGA2 includes a Distance (flat) transformation. This assigns a constant resistance of 1 to every cell, making the effective distance equivalent to Euclidean distance (for costDistance) or a random-walk equivalent.

The Distance transformation serves as a model for Isolation by Distance (IBD): genetic differentiation increases with geographic distance, with no heterogeneous landscape influence. Including it in SS_optim() (via dist_mod = TRUE) provides an important baseline. If geographic distance explains genetic patterns as well as any resistance surface, there is no evidence for landscape-mediated connectivity.

# The Distance transformation is fit automatically when dist_mod = TRUE in SS_optim()
ss_out <- SS_optim(
  gdist.inputs = gdist_inputs,
  GA.inputs    = ga_inputs,
  dist_mod     = TRUE,   # adds flat-distance IBD model
  null_mod     = TRUE    # adds intercept-only null model
)

Choosing Transformation Families

# For a single continuous layer: Monomolecular only
ga_inputs <- GA.prep(
  raster       = raster_orig,
  Results.dir  = file.path(tempdir(), "rga2-mono"),
  select.trans = list("M")
)

# For two continuous layers: Monomolecular only for both
ga_inputs_2 <- GA.prep(
  raster       = terra::subset(surfaces, c("continuous", "feature")),
  Results.dir  = file.path(tempdir(), "rga2-mono-2"),
  select.trans = list("M", "M")
)

# Include all 8 families for one layer
ga_inputs_all <- GA.prep(
  raster       = raster_orig,
  Results.dir  = file.path(tempdir(), "rga2-all"),
  select.trans = list("A")   # "A" = all 8 families
)

# Monomolecular for one layer, all families for another
ga_inputs_mixed <- GA.prep(
  raster       = terra::subset(surfaces, c("continuous", "feature")),
  Results.dir  = file.path(tempdir(), "rga2-mixed"),
  select.trans = list("M", "A")
)

ga_inputs_check <- GA.prep(
  raster      = terra::subset(surfaces, c("categorical", "continuous")),
  Results.dir = file.path(tempdir(), "rga2-check"),
  monitor     = FALSE,
  quiet       = TRUE
)

ga_inputs_check$select.trans
#> [[1]]
#> [1] NA
#> 
#> [[2]]
#> [1] 1 3 5 7 9
ga_inputs_check$surface.type
#> [1] "cat"  "cont"

Applying Transformations Directly

# Monomolecular transformation
r_mono <- Resistance.tran(
  transformation = "Monomolecular",
  shape = 3,
  max   = 150,
  r     = raster_orig[["cont_orig"]]
)

# Ricker transformation
r_rick <- Resistance.tran(
  transformation = "Ricker",
  shape = 2,
  max   = 100,
  r     = raster_orig[["cont_orig"]]
)

# Reverse Monomolecular (resistance decreases with covariate)
r_revmono <- Resistance.tran(
  transformation = "Reverse Monomolecular",
  shape = 2,
  max   = 200,
  r     = raster_orig[["cont_orig"]]
)

# Compare visually
compare_stack <- c(r_mono, r_rick, r_revmono)
names(compare_stack) <- c("Monomolecular", "Ricker", "Rev. Monomolecular")
plot(compare_stack)

Plot.trans()

Plot.trans() shows the transformation curve alongside a marginal histogram of the original raster values, making it easy to see how the transformation maps the covariate distribution to the resistance scale.

Plot.trans(
  PARM           = c(2, 100),
  Resistance     = raster_orig[["cont_orig"]],
  transformation = "Monomolecular",
  marginal.plot  = FALSE
)

Monomolecular (shape=2, max=100) showing the transformation curve and the distribution of original values.

---

How the GA Searches Transformation Space

During optimization, the genetic algorithm encodes each candidate solution as a vector of parameters. For a single continuous surface, this vector contains:

1. An integer index selecting the transformation family (from the families in select.trans).
2. The shape parameter (searched over a range, typically 0.5–15).
3. The max parameter (searched over a range, typically 1–500 or 1–1000).

For categorical surfaces, the vector contains one resistance value per class (searched over a range, typically 1–2500 per class).

For multi-surface models, the vector concatenates the parameter blocks for all surfaces plus a weight vector.

The GA does not evaluate all points on a grid. Instead, it maintains a population of candidate vectors, evaluates fitness (log-likelihood), and uses crossover and mutation to move toward high-fitness regions. This makes it effective at finding good solutions in high-dimensional continuous parameter spaces without exhaustive enumeration.

Note: Because the GA is stochastic, results can vary slightly across runs. This is expected and not a sign of failure; it reflects natural variation in exploration paths. Consistent results across replicate runs indicate that the optimization has converged to a stable region.

Overfitting Concerns

The more transformation families you include in the search, the more flexible the model and the greater the risk of fitting noise rather than signal. Key signs of potential overfitting:

1. Inconsistent results across replicates. If different GA runs return very different transformation families or very different parameter values, the optimization is not converging to a stable solution.

2. Very high max values. A max of 2000 for a dataset where pairwise distances span a small range may indicate the optimizer is exploiting numerical precision rather than biology.

3. Ricker or Monomolecular families with extreme shape values. Very large or very small shape parameters for these curves curves can produce near-flat responses that differ little from a Distance fit but use more parameters.

4. Better fit than biologically plausible. If you have extremely high \(R^2\) values, scrutinize the results carefully. Be suspect of such high explanatory power unless the sampling design, genetic markers, and landscape variables are all exceptionally well-matched.

Recommendation: Always run at least two GA replicates. If the results are highly consistent, you can have more confidence in the stability of the transformation selection. If they differ substantially, consider restricting select.trans to simpler families or increasing pop.size and maxiter.

Summary of Transformation Families

Quick reference: choosing and applying transformations

library(ResistanceGA2)
library(terra)

# 1. Inspect the raw resistance surface.
surface <- terra::unwrap(resistance_surfaces)[["continuous"]]
terra::plot(surface)

# 2. Preview a candidate transformation before optimization.
preview_surface <- Resistance.tran(
  transformation = "Monomolecular",
  shape = 2,
  max = 100,
  r = surface
)
terra::plot(preview_surface)

# 3. Plot the response curve to see how raw values map to resistance.
Plot.trans(
  PARM = c(2, 100),
  Resistance = surface,
  transformation = "Monomolecular",
  marginal.plot = FALSE
)

# 4. Restrict GA.prep() to transformation families with biological support.
ga_inputs <- GA.prep(
  raster = surface,
  Results.dir = file.path(tempdir(), "rga2-transformations"),
  select.trans = list(c("M", "RM"))
)

How to read the output: Resistance.tran() returns the transformed raster that would be passed to the distance engine. Plot.trans() shows the mapping from original raster values to optimized resistance values. Use that curve to ask whether the transformation matches the ecological hypothesis before letting the GA search it. A simpler select.trans set is easier to interpret and reduces the chance that the GA finds a flexible but biologically weak solution.

Summary of key functions

See also

• vignette("GettingStarted", package = "ResistanceGA2"), basic data setup and first optimization.
• vignette("ResistanceGA", package = "ResistanceGA2"), full optimization and model-comparison workflow.
• vignette("IBE_IBR_Framework", package = "ResistanceGA2"), using transformed resistance surfaces in conditional IBE/IBR models.

References

Clarke, R. T., P. Rothery, and A. F. Raybould. 2002. Confidence limits for regression relationships between distance matrices: Estimating gene flow with distance. Journal of Agricultural, Biological, and Environmental Statistics 7:361-372. https://doi.org/10.1198/108571102320

Peterman, W. E. 2018. ResistanceGA: An R package for the optimization of resistance surfaces using genetic algorithms. Methods in Ecology and Evolution 9:1638-1647. https://doi.org/10.1111/2041-210X.12984

Scrucca, L. 2013. GA: A package for genetic algorithms in R. Journal of Statistical Software 53:1–37.

Beninde, J., J. Wittische, and A. C. Frantz. 2024. Quantifying uncertainty in inferences of landscape genetic resistance due to choice of individual-based genetic distance metric. Molecular Ecology Resources 24:e13831.