---
title: "Large Landscapes"
author: "Bill Peterman"
output:
rmarkdown::html_vignette:
number_sections: true
toc: true
self_contained: true
mathjax: null
pandoc_args: ["--mathml"]
vignette: >
%\VignetteIndexEntry{Large Landscapes}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r setup, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.width = 6,
fig.height = 4.25,
warning = FALSE,
message = FALSE
)
```
This vignette covers fitting `terradish` models on large rasters, with
hundreds of thousands to millions of grid cells. It assumes you can already
fit and read a small model as shown in
[`vignette("getting-started", package = "terradish")`](getting-started.html).
The runnable setup uses the Ohio pond data from the other guides
(`vignette("data-generation", package = "terradish")` describes how they
were made). It teaches configuration, not speed. Time and memory depend on
graph geometry, conductance contrast, focal-site count, and hardware, so
measure them on your own design.
**Goal.** Start with solver choice and domain sensitivity. Read the optional
sections on worker reuse, nuisance optimization, and curvature when a
measured computational problem calls for them.
```{r load}
library(terradish)
data(ohio_ponds)
coords <- ohio_ponds$ponds[c("x", "y")]
g <- factor(ohio_ponds$samples$pond, levels = ohio_ponds$ponds$pond)
Y <- rowsum(ohio_ponds$snp, g, reorder = FALSE)
Y <- Y[ohio_ponds$ponds$pond, , drop = FALSE]
N <- 2 * as.numeric(table(g))
stopifnot(all(N == N[1]))
S <- cov_from_biallelic(Y, N) / N[1]
covariates <- scale_covariates(terra::unwrap(ohio_ponds$forest500))
surface <- conductance_surface(covariates, coords, directions = 8)
```
The graph covers the full landscape, but the response is covariance among 24
ponds, so more raster cells add no sampled populations or marker information.
The fitting recipes below use the `nu = 40` working assumption from Getting
Started and do not run during package builds.
```{r domain-map, fig.cap = "The full forest-context grid and sampled ponds. Computational shortcuts must preserve the routes needed for the biological question."}
terra::plot(covariates, main = "Standardized 500 m forest context")
points(coords, pch = 21, bg = "white", cex = 0.7)
```
**How to read the map.** Sampled ponds occupy only part of the raster, but
paths connecting them can extend beyond it. A tightly cropped map is
therefore not automatically an equivalent, faster analysis (see the cropping
section below).
## When a landscape is "large"
`terradish` builds a weighted graph over the raster grid and solves a linear
system on it to get resistance distances among focal sites. The cost of that
solve grows with the number of grid cells, `N`. In the hundreds of thousands
and beyond:
1. The linear **solver** can run out of memory, so solver choice decides
whether a fit is feasible.
2. The **inference** machinery (standard errors and confidence intervals)
keeps its form, but larger analyses can include more focal sites, and
Wishart precision still depends on the supplied effective degrees of
freedom, `nu` (see the information section below).
"Large" means that graph factorization or repeated derivative solves dominate
the analysis. There is no universal cell-count limit, because missing cells,
graph connectivity, and conductance contrast can change the practical limit.
---
## Choosing a solver
Both `terradish()` (the fitting wrapper) and `terradish_algorithm()` (one
likelihood and gradient evaluation) take a `solver` argument. The solver
changes **how** the resistance-distance system is solved, not the statistical
model. Iterative solvers should agree with the direct solver within numerical
tolerance, though not necessarily bit for bit.
```{r solver-calls, eval = FALSE}
# direct (the default): exact sparse Cholesky factorization
fit <- terradish(S ~ forest500, surface,
measurement_model = wishart_covariance, nu = 40,
solver = "direct")
# amg: algebraic multigrid, the large-N path
fit <- terradish(S ~ forest500, surface,
measurement_model = wishart_covariance, nu = 40,
solver = "amg")
```
The choices are:
- **`direct`** (default) uses sparse Cholesky factorization. The factor can
hold many more nonzero entries than the graph, so its memory can grow
rapidly.
- **`amg`** uses algebraic multigrid, an iterative solver that can reduce
memory on large grids. Its convergence depends on conductance contrast and
graph structure, and strong barriers can require more iterations or tighter
solver settings.
- **`auto`**, with default controls, uses `direct` below 1,500,000 graph
vertices and `amg` at or above. The number of focal sites does not override
this switch, and `solver_control$auto_amg_min_vertices` changes the threshold.
Before a long fit, `terradish_assess_settings()` runs short optimizer and
solver probes on your graph and recommends settings, and
`terradish_solver_benchmark()` times each direct factorization:
```{r assess-settings, eval = FALSE}
settings <- terradish_assess_settings(S ~ forest500, surface,
measurement_model = wishart_covariance, nu = 40)
settings # graph size, probe timings, and a one-line verdict
settings$recommended$solver # pass to terradish(solver = ...)
settings$recommended$optimizer # pass to terradish(optimizer = ...)
terradish_solver_benchmark(surface) # setup, solve, and total seconds per factorization
```
**How to read the output.** `settings$profile` reports graph size, the
benchmark tables give each probe's time and whether it reached the same
log-likelihood, and `settings$recommended` holds settings to pass straight to
`terradish()`. The fastest `total` time in the solver benchmark names the
factorization to request with `solver_control = list(factorization = ...)`.
On a graph this small the timings mostly measure R overhead, so run both on
your full raster. The recommendation concerns speed and agreement, not whether
the measurement model suits your data.
Solver choice applies to every symmetric resistance model, whatever the
`conductance_model` (`loglinear_conductance`, `smooth_loglinear_conductance`,
the Gaussian scale-aware model) or measurement model (`mlpe`,
`leastsquares`, `generalized_wishart`, `wishart_covariance`).
After a fit, use the same diagnostics as for a small graph, starting with the
fitted-response scatter, and keep the fitted graph for conductance mapping.
```{r large-fit-plots, eval = FALSE}
plot(fit, type = "fit") # observed versus fitted covariance for this response
library(ggplot2)
maps <- plot(fit, type = "surface", data = surface)
C <- conductance(surface, fit)
map_limits <- range(terra::values(C), na.rm = TRUE)
# Keep all cells and share limits across the estimate and both bounds.
for (p in maps) {
print(p + scale_fill_viridis_c(
trans = "log10", limits = map_limits,
breaks = c(0.01, 0.1, 0.5, 1, 2, 5, 10, 20),
name = "Conductance\n(log color scale)"
))
}
```
Map colors run from dark purple (low conductance) to yellow (high). Equal
color steps are equal ratios, labels stay in conductance units, and the
estimate and bounds share limits without clipping.
A smooth map does not establish solver accuracy. Check that estimates and
likelihoods agree with a direct solve on a manageable case, and inspect both
outer and nuisance convergence. Marginal-response plots need a graph solve at
every plotted value, so start with `n = 25` on a large graph.
---
Optional: worker reuse, nuisance optimization, and curvature
## Reusing Windows workers
Worker reuse is **experimental**. On Windows, `cores > 1` requests PSOCK
workers for supported Hessian and partial-derivative calculations, not for
every step of a fit. The pool is reused during optimization and stopped before
return, so fitted objects hold no live workers. AMG and the cached CHOLMOD
backend ignore this setting. On small graphs, worker startup and data transfer
can make multiple cores slower than one, so benchmark `cores = 1` first and
compare elapsed time and results with the requested worker count.
Use one level of parallelism at a time. Fixed-domain cross-validation, for
example, evaluates folds serially and passes `cores` to each fit.
## Controlling the nuisance profile
Each trial of the conductance parameters profiles the measurement model's
nuisance parameters. Smooth conductance models paired with
`generalized_wishart` can make this two-level optimization costly, because
each outer step may need many inner Newton steps. The inner optimizer already
warm-starts from the previous nuisance solution. Use `measurement_control` to
inspect or tune it separately from `control`:
```{r measurement-control, eval = FALSE}
D <- dist_from_cov(S) # coherent squared distances from the pond covariance
fit_smooth_gw <- terradish(
D ~ s(forest500, df = 3),
surface,
conductance_model = smooth_loglinear_conductance,
measurement_model = generalized_wishart,
nu = 40,
optimizer = "bfgs",
control = NewtonRaphsonControl(maxit = 80, verbose = TRUE),
measurement_control = NewtonRaphsonControl(
maxit = 40,
ctol = 1e-8,
ftol = 1e-10,
verbose = FALSE
)
)
fit_smooth_gw$fit$subproblem
```
The returned `convergence` code and `iters` count describe the final nuisance
profile. A code of 0 means convergence, 1 means the profile reached `maxit`,
and 2 means a line-search failure. Do not loosen tolerances just to remove a
warning. First inspect coefficient stability, gradient size, the likelihood,
and sensitivity to tighter settings, because overly loose profiling can change
the fitted likelihood surface.
## Curvature for standard errors
`terradish()` and `terradish_algorithm()` take `curvature = c("exact",
"gauss_newton")`. Both options share the objective and gradient and differ
only in the Hessian used for optimization and standard errors.
```{r curvature-calls, eval = FALSE}
D <- dist_from_cov(S)
# Generalized Wishart requires an admissible squared-distance response that is
# coherently related to a centered positive-semidefinite covariance matrix.
# exact (default): full observed Hessian
fit <- terradish(D ~ forest500, surface,
measurement_model = generalized_wishart,
nu = 40,
curvature = "exact")
# gauss_newton: information-based curvature
fit <- terradish(D ~ forest500, surface,
measurement_model = generalized_wishart,
nu = 40,
curvature = "gauss_newton")
```
- **`exact`** uses the full observed Hessian.
- **`gauss_newton`** drops the two residual-weighted second-derivative terms,
leaving the **Gauss-Newton information** curvature. It needs only first
derivatives of conductance and is positive semidefinite when the retained
measurement-model curvature is. It flows unchanged into `vcov()`,
`summary()`, and `confint()`, so standard errors become asymptotic,
information-based errors. It approaches the exact Hessian when the omitted
terms are negligible, not merely because a fit has converged.
**How to choose.** Both are validated for `leastsquares` and
`generalized_wishart`. Use the `exact` default for routine work. Reach for
`gauss_newton` when the optimizer struggles with an indefinite Hessian far
from the optimum, or when you want strictly information-based standard
errors. The two may agree closely at a well-behaved fit, but compare them as
a sensitivity check when curvature choice materially changes uncertainty or
convergence.
The benchmark in `inst/benchmarks/benchmark-curvature-large-landscape.R`
reproduces this comparison. It simulates a `generalized_wishart`
squared-distance response from known conductance parameters on a synthetic
heterogeneous raster, then fits the same model with each curvature option.
```{r curvature-benchmark-run, eval = FALSE}
source(system.file("benchmarks", "benchmark-curvature-large-landscape.R",
package = "terradish"))
bench <- run_curvature_large_landscape_benchmark(CONFIG)
bench$comparison
```
The default case is small enough to run during review. For the large-N solver
regime, copy `CONFIG`, increase `side`, and set `solver = "amg"` or
`solver = "auto"` before calling `run_curvature_large_landscape_benchmark()`.
**How to read the benchmark.** Compare coefficients and likelihoods first,
then standard errors, Hessian eigenvalues, and elapsed time, because matching
point estimates do not guarantee matching curvature. A large difference in
uncertainty warrants checking residuals, conditioning, and model
misspecification before choosing the faster option.
## Information and effective degrees of freedom
For Wishart fits, `nu` is an effective degrees-of-freedom value that you
supply, not automatically the marker count.
Changing `nu` leaves the optimum unchanged in theory and scales interior
standard errors in proportion to `1 / sqrt(nu)`, which `terradish_rescale_nu()`
shows without refitting. See
[`vignette("wishart-covariance", package = "terradish")`](wishart-covariance.html)
for choosing `nu` and
[`vignette("model-comparison", package = "terradish")`](model-comparison.html)
for reporting inference across plausible values.
## Cropping and coarse warm starts
Cropping removes alternative routes outside the retained domain and can
increase resistance between sites even when every site stays inside the crop.
In one test landscape, cropping raised resistance by 15 to 23% with a two-cell
buffer and by 1 to 2% with a ten-cell buffer. Values will differ on your
landscape, so compare progressively wider buffers with the same response,
sites, formula, and covariate scaling.
```{r crop-sensitivity, eval = FALSE}
# Scale on the original domain once, before cropping.
buffers <- c(2, 5, 10) * max(terra::res(covariates))
fits_buffer <- lapply(buffers, function(buffer) {
cropped_surface <- conductance_surface(
covariates, coords, directions = 8, crop_buffer = buffer
)
terradish(S ~ forest500, cropped_surface,
measurement_model = wishart_covariance, nu = 40)
})
lapply(fits_buffer, coef)
lapply(fits_buffer, confint)
# crop_to_focal_buffer() crops the raster stack directly, if you prefer to
# build the graph yourself; it keeps the scale_covariates() metadata.
cropped <- crop_to_focal_buffer(covariates, coords, buffer = 2000)
c(full = terra::ncell(covariates), cropped = terra::ncell(cropped))
```
**How to read the output.** Compare estimates and intervals across buffers.
If the coefficient or its interval keeps changing as the buffer widens, keep
widening until it stabilizes. `terradish_distance()` compares the fitted
resistance distances directly. Both cropping routes print a message when the
buffer is less than half the largest distance between sites. Treat it as a
prompt to run this check, not as proof that a larger buffer is adequate.
Coarse rasters can provide starting values, but coarsening changes both routes
and covariate summaries, so the supported workflow requires exact refinement
on the original graph:
```{r coarse-warm-start, eval = FALSE}
fit <- terradish(
S ~ forest500, surface,
measurement_model = wishart_covariance, nu = 40,
approximation = "coarse_raster",
approximation_control = list(factor = c(4, 2), exact_refine = TRUE)
)
summary(fit)
```
Inspect convergence after the final refinement, because a converged coarse fit
does not establish convergence on the original graph. `terradish_grid()`
accepts only `approximation = "none"`, and coarse-only likelihoods are not
supported for ranking models.
---
## Storing large fitted objects
Fitted conductance and measurement-model closures can retain their
construction environments, including large graph objects. That helps
prediction but can make `saveRDS()` files much larger than the coefficient
table suggests. Use `slim_terradish()` when the saved object is for inference
summaries rather than later prediction:
```{r slim-fit, eval = FALSE}
# fit is the unslimmed model you have already fitted above.
fit_slim <- slim_terradish(fit)
fit_slim$storage
saveRDS(fit_slim, file.path(tempdir(), "terradish-fit-slim.rds"))
# Equivalent shorthand at fit time
D <- dist_from_cov(S)
fit_slim <- terradish(
D ~ forest500,
surface,
measurement_model = generalized_wishart,
nu = 40,
slim = TRUE
)
```
The default removes `submodels` and `leverage`. Coefficients, likelihood,
Hessian-based summaries, fitted values, diagnostics, and permutation
simulation remain available. Conductance prediction and most plot types do not,
because they must reevaluate the removed closures. Slimming is irreversible, so
keep the formula, graph, response, and an unslimmed fit whenever you may need
prediction or model reconstruction. Add `"response"` to `drop` only when the
response must be excluded from the archive, because that also removes the
strongest stored check for later model comparison.
---
Quick reference: computational settings
## Quick reference
```{r quick-ref, eval = FALSE}
# 1. Use the pond covariance and landscape from the setup above
D <- dist_from_cov(S)
surface <- conductance_surface(covariates, coords, directions = 8)
# 2. Fit at scale: use amg once the raster is too large for the direct solver
# Generalized Wishart requires an admissible squared-distance response.
fit <- terradish(D ~ forest500, surface,
conductance_model = loglinear_conductance,
measurement_model = generalized_wishart,
nu = 40,
solver = "amg", # large-N linear solver
curvature = "gauss_newton") # information-based standard errors
summary(fit) # conditional estimates and SEs
# Optional archival copy when prediction from the saved object is unnecessary
fit_archive <- slim_terradish(fit)
saveRDS(fit_archive, file.path(tempdir(), "fit-archive.rds"))
```
## Summary of key functions
| Function | Purpose |
|----------|---------|
| `conductance_surface()` | Build the weighted graph over the raster grid |
| `terradish()` | Fit the model; takes `solver` and `curvature` |
| `terradish_algorithm()` | One likelihood and gradient evaluation; same `solver` and `curvature` |
| `crop_to_focal_buffer()` | Crop with an explicit map-unit buffer; check domain sensitivity |
| `terradish_assess_settings()` | Compare computational settings on the current graph |
| `terradish_solver_benchmark()` | Time each direct factorization on the current graph |
| `slim_terradish()` | Remove prediction closures and other optional large components before storage |
## See also
- **[`vignette("getting-started", package = "terradish")`](getting-started.html)**: the core workflow,
from data to interpretation.
- **[`vignette("wishart-covariance", package = "terradish")`](wishart-covariance.html)**: the Wishart
measurement models and the role of `nu`.
- **[`vignette("model-comparison", package = "terradish")`](model-comparison.html)**: ranking competing
models with AIC and likelihood ratio tests.
- **[`vignette("gaussian-scale-optimization", package = "terradish")`](gaussian-scale-optimization.html)**: Gaussian
scale models, which also accept coarse-raster warm starts.
- **[`vignette("simulation-design", package = "terradish")`](simulation-design.html)**: design sweeps
that require many fits on the same graph.