--- title: "Large Landscapes" author: "Bill Peterman" output: rmarkdown::html_vignette: number_sections: true toc: true self_contained: true mathjax: null pandoc_args: ["--mathml"] vignette: > %\VignetteIndexEntry{Large Landscapes} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4.25, warning = FALSE, message = FALSE ) ``` This vignette covers fitting `terradish` models on large rasters, with hundreds of thousands to millions of grid cells. It assumes you can already fit and read a small model as shown in [`vignette("getting-started", package = "terradish")`](getting-started.html). The runnable setup uses the Ohio pond data from the other guides (`vignette("data-generation", package = "terradish")` describes how they were made). It teaches configuration, not speed. Time and memory depend on graph geometry, conductance contrast, focal-site count, and hardware, so measure them on your own design. **Goal.** Start with solver choice and domain sensitivity. Read the optional sections on worker reuse, nuisance optimization, and curvature when a measured computational problem calls for them. ```{r load} library(terradish) data(ohio_ponds) coords <- ohio_ponds$ponds[c("x", "y")] g <- factor(ohio_ponds$samples$pond, levels = ohio_ponds$ponds$pond) Y <- rowsum(ohio_ponds$snp, g, reorder = FALSE) Y <- Y[ohio_ponds$ponds$pond, , drop = FALSE] N <- 2 * as.numeric(table(g)) stopifnot(all(N == N[1])) S <- cov_from_biallelic(Y, N) / N[1] covariates <- scale_covariates(terra::unwrap(ohio_ponds$forest500)) surface <- conductance_surface(covariates, coords, directions = 8) ``` The graph covers the full landscape, but the response is covariance among 24 ponds, so more raster cells add no sampled populations or marker information. The fitting recipes below use the `nu = 40` working assumption from Getting Started and do not run during package builds. ```{r domain-map, fig.cap = "The full forest-context grid and sampled ponds. Computational shortcuts must preserve the routes needed for the biological question."} terra::plot(covariates, main = "Standardized 500 m forest context") points(coords, pch = 21, bg = "white", cex = 0.7) ``` **How to read the map.** Sampled ponds occupy only part of the raster, but paths connecting them can extend beyond it. A tightly cropped map is therefore not automatically an equivalent, faster analysis (see the cropping section below). ## When a landscape is "large" `terradish` builds a weighted graph over the raster grid and solves a linear system on it to get resistance distances among focal sites. The cost of that solve grows with the number of grid cells, `N`. In the hundreds of thousands and beyond: 1. The linear **solver** can run out of memory, so solver choice decides whether a fit is feasible. 2. The **inference** machinery (standard errors and confidence intervals) keeps its form, but larger analyses can include more focal sites, and Wishart precision still depends on the supplied effective degrees of freedom, `nu` (see the information section below). "Large" means that graph factorization or repeated derivative solves dominate the analysis. There is no universal cell-count limit, because missing cells, graph connectivity, and conductance contrast can change the practical limit. --- ## Choosing a solver Both `terradish()` (the fitting wrapper) and `terradish_algorithm()` (one likelihood and gradient evaluation) take a `solver` argument. The solver changes **how** the resistance-distance system is solved, not the statistical model. Iterative solvers should agree with the direct solver within numerical tolerance, though not necessarily bit for bit. ```{r solver-calls, eval = FALSE} # direct (the default): exact sparse Cholesky factorization fit <- terradish(S ~ forest500, surface, measurement_model = wishart_covariance, nu = 40, solver = "direct") # amg: algebraic multigrid, the large-N path fit <- terradish(S ~ forest500, surface, measurement_model = wishart_covariance, nu = 40, solver = "amg") ``` The choices are: - **`direct`** (default) uses sparse Cholesky factorization. The factor can hold many more nonzero entries than the graph, so its memory can grow rapidly. - **`amg`** uses algebraic multigrid, an iterative solver that can reduce memory on large grids. Its convergence depends on conductance contrast and graph structure, and strong barriers can require more iterations or tighter solver settings. - **`auto`**, with default controls, uses `direct` below 1,500,000 graph vertices and `amg` at or above. The number of focal sites does not override this switch, and `solver_control$auto_amg_min_vertices` changes the threshold. Before a long fit, `terradish_assess_settings()` runs short optimizer and solver probes on your graph and recommends settings, and `terradish_solver_benchmark()` times each direct factorization: ```{r assess-settings, eval = FALSE} settings <- terradish_assess_settings(S ~ forest500, surface, measurement_model = wishart_covariance, nu = 40) settings # graph size, probe timings, and a one-line verdict settings$recommended$solver # pass to terradish(solver = ...) settings$recommended$optimizer # pass to terradish(optimizer = ...) terradish_solver_benchmark(surface) # setup, solve, and total seconds per factorization ``` **How to read the output.** `settings$profile` reports graph size, the benchmark tables give each probe's time and whether it reached the same log-likelihood, and `settings$recommended` holds settings to pass straight to `terradish()`. The fastest `total` time in the solver benchmark names the factorization to request with `solver_control = list(factorization = ...)`. On a graph this small the timings mostly measure R overhead, so run both on your full raster. The recommendation concerns speed and agreement, not whether the measurement model suits your data. Solver choice applies to every symmetric resistance model, whatever the `conductance_model` (`loglinear_conductance`, `smooth_loglinear_conductance`, the Gaussian scale-aware model) or measurement model (`mlpe`, `leastsquares`, `generalized_wishart`, `wishart_covariance`). After a fit, use the same diagnostics as for a small graph, starting with the fitted-response scatter, and keep the fitted graph for conductance mapping. ```{r large-fit-plots, eval = FALSE} plot(fit, type = "fit") # observed versus fitted covariance for this response library(ggplot2) maps <- plot(fit, type = "surface", data = surface) C <- conductance(surface, fit) map_limits <- range(terra::values(C), na.rm = TRUE) # Keep all cells and share limits across the estimate and both bounds. for (p in maps) { print(p + scale_fill_viridis_c( trans = "log10", limits = map_limits, breaks = c(0.01, 0.1, 0.5, 1, 2, 5, 10, 20), name = "Conductance\n(log color scale)" )) } ``` Map colors run from dark purple (low conductance) to yellow (high). Equal color steps are equal ratios, labels stay in conductance units, and the estimate and bounds share limits without clipping. A smooth map does not establish solver accuracy. Check that estimates and likelihoods agree with a direct solve on a manageable case, and inspect both outer and nuisance convergence. Marginal-response plots need a graph solve at every plotted value, so start with `n = 25` on a large graph. ---
Optional: worker reuse, nuisance optimization, and curvature ## Reusing Windows workers Worker reuse is **experimental**. On Windows, `cores > 1` requests PSOCK workers for supported Hessian and partial-derivative calculations, not for every step of a fit. The pool is reused during optimization and stopped before return, so fitted objects hold no live workers. AMG and the cached CHOLMOD backend ignore this setting. On small graphs, worker startup and data transfer can make multiple cores slower than one, so benchmark `cores = 1` first and compare elapsed time and results with the requested worker count. Use one level of parallelism at a time. Fixed-domain cross-validation, for example, evaluates folds serially and passes `cores` to each fit. ## Controlling the nuisance profile Each trial of the conductance parameters profiles the measurement model's nuisance parameters. Smooth conductance models paired with `generalized_wishart` can make this two-level optimization costly, because each outer step may need many inner Newton steps. The inner optimizer already warm-starts from the previous nuisance solution. Use `measurement_control` to inspect or tune it separately from `control`: ```{r measurement-control, eval = FALSE} D <- dist_from_cov(S) # coherent squared distances from the pond covariance fit_smooth_gw <- terradish( D ~ s(forest500, df = 3), surface, conductance_model = smooth_loglinear_conductance, measurement_model = generalized_wishart, nu = 40, optimizer = "bfgs", control = NewtonRaphsonControl(maxit = 80, verbose = TRUE), measurement_control = NewtonRaphsonControl( maxit = 40, ctol = 1e-8, ftol = 1e-10, verbose = FALSE ) ) fit_smooth_gw$fit$subproblem ``` The returned `convergence` code and `iters` count describe the final nuisance profile. A code of 0 means convergence, 1 means the profile reached `maxit`, and 2 means a line-search failure. Do not loosen tolerances just to remove a warning. First inspect coefficient stability, gradient size, the likelihood, and sensitivity to tighter settings, because overly loose profiling can change the fitted likelihood surface. ## Curvature for standard errors `terradish()` and `terradish_algorithm()` take `curvature = c("exact", "gauss_newton")`. Both options share the objective and gradient and differ only in the Hessian used for optimization and standard errors. ```{r curvature-calls, eval = FALSE} D <- dist_from_cov(S) # Generalized Wishart requires an admissible squared-distance response that is # coherently related to a centered positive-semidefinite covariance matrix. # exact (default): full observed Hessian fit <- terradish(D ~ forest500, surface, measurement_model = generalized_wishart, nu = 40, curvature = "exact") # gauss_newton: information-based curvature fit <- terradish(D ~ forest500, surface, measurement_model = generalized_wishart, nu = 40, curvature = "gauss_newton") ``` - **`exact`** uses the full observed Hessian. - **`gauss_newton`** drops the two residual-weighted second-derivative terms, leaving the **Gauss-Newton information** curvature. It needs only first derivatives of conductance and is positive semidefinite when the retained measurement-model curvature is. It flows unchanged into `vcov()`, `summary()`, and `confint()`, so standard errors become asymptotic, information-based errors. It approaches the exact Hessian when the omitted terms are negligible, not merely because a fit has converged. **How to choose.** Both are validated for `leastsquares` and `generalized_wishart`. Use the `exact` default for routine work. Reach for `gauss_newton` when the optimizer struggles with an indefinite Hessian far from the optimum, or when you want strictly information-based standard errors. The two may agree closely at a well-behaved fit, but compare them as a sensitivity check when curvature choice materially changes uncertainty or convergence. The benchmark in `inst/benchmarks/benchmark-curvature-large-landscape.R` reproduces this comparison. It simulates a `generalized_wishart` squared-distance response from known conductance parameters on a synthetic heterogeneous raster, then fits the same model with each curvature option. ```{r curvature-benchmark-run, eval = FALSE} source(system.file("benchmarks", "benchmark-curvature-large-landscape.R", package = "terradish")) bench <- run_curvature_large_landscape_benchmark(CONFIG) bench$comparison ``` The default case is small enough to run during review. For the large-N solver regime, copy `CONFIG`, increase `side`, and set `solver = "amg"` or `solver = "auto"` before calling `run_curvature_large_landscape_benchmark()`. **How to read the benchmark.** Compare coefficients and likelihoods first, then standard errors, Hessian eigenvalues, and elapsed time, because matching point estimates do not guarantee matching curvature. A large difference in uncertainty warrants checking residuals, conditioning, and model misspecification before choosing the faster option.
## Information and effective degrees of freedom For Wishart fits, `nu` is an effective degrees-of-freedom value that you supply, not automatically the marker count. Changing `nu` leaves the optimum unchanged in theory and scales interior standard errors in proportion to `1 / sqrt(nu)`, which `terradish_rescale_nu()` shows without refitting. See [`vignette("wishart-covariance", package = "terradish")`](wishart-covariance.html) for choosing `nu` and [`vignette("model-comparison", package = "terradish")`](model-comparison.html) for reporting inference across plausible values. ## Cropping and coarse warm starts Cropping removes alternative routes outside the retained domain and can increase resistance between sites even when every site stays inside the crop. In one test landscape, cropping raised resistance by 15 to 23% with a two-cell buffer and by 1 to 2% with a ten-cell buffer. Values will differ on your landscape, so compare progressively wider buffers with the same response, sites, formula, and covariate scaling. ```{r crop-sensitivity, eval = FALSE} # Scale on the original domain once, before cropping. buffers <- c(2, 5, 10) * max(terra::res(covariates)) fits_buffer <- lapply(buffers, function(buffer) { cropped_surface <- conductance_surface( covariates, coords, directions = 8, crop_buffer = buffer ) terradish(S ~ forest500, cropped_surface, measurement_model = wishart_covariance, nu = 40) }) lapply(fits_buffer, coef) lapply(fits_buffer, confint) # crop_to_focal_buffer() crops the raster stack directly, if you prefer to # build the graph yourself; it keeps the scale_covariates() metadata. cropped <- crop_to_focal_buffer(covariates, coords, buffer = 2000) c(full = terra::ncell(covariates), cropped = terra::ncell(cropped)) ``` **How to read the output.** Compare estimates and intervals across buffers. If the coefficient or its interval keeps changing as the buffer widens, keep widening until it stabilizes. `terradish_distance()` compares the fitted resistance distances directly. Both cropping routes print a message when the buffer is less than half the largest distance between sites. Treat it as a prompt to run this check, not as proof that a larger buffer is adequate. Coarse rasters can provide starting values, but coarsening changes both routes and covariate summaries, so the supported workflow requires exact refinement on the original graph: ```{r coarse-warm-start, eval = FALSE} fit <- terradish( S ~ forest500, surface, measurement_model = wishart_covariance, nu = 40, approximation = "coarse_raster", approximation_control = list(factor = c(4, 2), exact_refine = TRUE) ) summary(fit) ``` Inspect convergence after the final refinement, because a converged coarse fit does not establish convergence on the original graph. `terradish_grid()` accepts only `approximation = "none"`, and coarse-only likelihoods are not supported for ranking models. --- ## Storing large fitted objects Fitted conductance and measurement-model closures can retain their construction environments, including large graph objects. That helps prediction but can make `saveRDS()` files much larger than the coefficient table suggests. Use `slim_terradish()` when the saved object is for inference summaries rather than later prediction: ```{r slim-fit, eval = FALSE} # fit is the unslimmed model you have already fitted above. fit_slim <- slim_terradish(fit) fit_slim$storage saveRDS(fit_slim, file.path(tempdir(), "terradish-fit-slim.rds")) # Equivalent shorthand at fit time D <- dist_from_cov(S) fit_slim <- terradish( D ~ forest500, surface, measurement_model = generalized_wishart, nu = 40, slim = TRUE ) ``` The default removes `submodels` and `leverage`. Coefficients, likelihood, Hessian-based summaries, fitted values, diagnostics, and permutation simulation remain available. Conductance prediction and most plot types do not, because they must reevaluate the removed closures. Slimming is irreversible, so keep the formula, graph, response, and an unslimmed fit whenever you may need prediction or model reconstruction. Add `"response"` to `drop` only when the response must be excluded from the archive, because that also removes the strongest stored check for later model comparison. ---
Quick reference: computational settings ## Quick reference ```{r quick-ref, eval = FALSE} # 1. Use the pond covariance and landscape from the setup above D <- dist_from_cov(S) surface <- conductance_surface(covariates, coords, directions = 8) # 2. Fit at scale: use amg once the raster is too large for the direct solver # Generalized Wishart requires an admissible squared-distance response. fit <- terradish(D ~ forest500, surface, conductance_model = loglinear_conductance, measurement_model = generalized_wishart, nu = 40, solver = "amg", # large-N linear solver curvature = "gauss_newton") # information-based standard errors summary(fit) # conditional estimates and SEs # Optional archival copy when prediction from the saved object is unnecessary fit_archive <- slim_terradish(fit) saveRDS(fit_archive, file.path(tempdir(), "fit-archive.rds")) ```
## Summary of key functions | Function | Purpose | |----------|---------| | `conductance_surface()` | Build the weighted graph over the raster grid | | `terradish()` | Fit the model; takes `solver` and `curvature` | | `terradish_algorithm()` | One likelihood and gradient evaluation; same `solver` and `curvature` | | `crop_to_focal_buffer()` | Crop with an explicit map-unit buffer; check domain sensitivity | | `terradish_assess_settings()` | Compare computational settings on the current graph | | `terradish_solver_benchmark()` | Time each direct factorization on the current graph | | `slim_terradish()` | Remove prediction closures and other optional large components before storage | ## See also - **[`vignette("getting-started", package = "terradish")`](getting-started.html)**: the core workflow, from data to interpretation. - **[`vignette("wishart-covariance", package = "terradish")`](wishart-covariance.html)**: the Wishart measurement models and the role of `nu`. - **[`vignette("model-comparison", package = "terradish")`](model-comparison.html)**: ranking competing models with AIC and likelihood ratio tests. - **[`vignette("gaussian-scale-optimization", package = "terradish")`](gaussian-scale-optimization.html)**: Gaussian scale models, which also accept coarse-raster warm starts. - **[`vignette("simulation-design", package = "terradish")`](simulation-design.html)**: design sweeps that require many fits on the same graph.