Using the cores you already have

Stage four: the machine you are sitting at, before any of the rest of it

Draft. The code runs and the argument holds; the prose has not been copy-edited.

The three stages before this one were about moving data. From here on the data has stopped being the problem and the repeats have started being the problem. This page is the first step and deliberately the least ambitious one: the cores in the session you are already sitting in.

It is a stage of its own because the choice of backend is made once and everything after it inherits the answer. Folded into the simulation, it would look like a detail of that simulation, and it is not.

library(dplyr)
library(tibble)
library(purrr)
library(glue)
library(cli)
library(parallel)

catchment_panel <- readRDS("catchment_panel.rds")

The repeats are the cost now, not the crossing

The table shrank by three orders of magnitude on Exploring and reducing the big table, and what came back fits comfortably in the session.

panel_kb <- as.numeric(object.size(catchment_panel)) / 1024

cli_alert_info(
  "{nrow(catchment_panel)} catchments, {round(panel_kb)} KB"
)
ℹ 810 catchments, 168 KB

Nothing after this point is limited by how much data can move. Everything after this point is limited by how many times a small computation has to run.

That changes which question is useful. Up to here it was where the collect() line sits. From here it is how many cores are turning.

What the shape has to be

The test, stated once and used by both simulation stages that follow: if repeat 200 needs nothing that repeat 199 produced, the work is this shape. Almost all simulation, bootstrapping and resampling is.

Both stages that follow satisfy it, which is why this page can settle the backend for both.

Try furrr first, and watch what happens

furrr::future_map() over a future::plan() is the natural first reach. It is one line of change from purrr::map(), and on a laptop it is the right answer.

c(future = "future", furrr = "furrr") |>
  map_lgl(\(p) requireNamespace(p, quietly = TRUE))
future  furrr 
  TRUE   TRUE

Whether those are present depends on your session. Installing them is ordinary here and costs a minute.

That is the whole story locally. It is not the story on the cluster, and the next section is where the difference bites.

The one thing that does not travel

This is the finding that shapes the rest of the example, and the reason this page comes before the simulations rather than inside one.

Asked on the cluster’s own workers, the answer is:

package on the worker?
dplyr yes
sf yes
parallel yes
furrr no

future and furrr are not in the runtime bundle. A plan that depends on them has nothing to load on the far side, and putting them there is an init-script change rather than an install.packages() call, which means asking someone and waiting. Base parallel is present in every context, including the worker.

So the two are not interchangeable, and the difference is not about which is better R:

  • In your own session, furrr once installed, because the ergonomics are better and the change from serial is one line.
  • Inside anything that might run on the cluster, parallel::mclapply(), because it is there.

This is a fact about the runtime as it stands, not a permanent property. An administrator can change it, and Packages is where that conversation belongs.

mclapply(), and why forking is the right default here

The version this example carries forward.

source("R/setup.R")
n_cores <- wq_cores()
n_cores
[1] 7

Forking is available because this is Linux, and it is cheaper than the alternative because the child starts as a copy-on-write view of the parent rather than a fresh copy. plan(multisession), and the equivalent socket cluster in parallel, is what to reach for on Windows and what not to reach for here.

Copy-on-write is worth being precise about, because the next section depends on it. It means a worker pays nothing to read what the parent already had. It does not mean a worker pays nothing for what it allocates itself, and a spatial index built inside the worker is exactly that.

Count the cores you are allowed, not the cores you can see

detectCores() is the obvious call and it answers the wrong question. It reports the machine’s cores, which is not the same as your share of them.

tibble(
  `detectCores()` = detectCores(),
  `cgroup quota` = wq_cpu_quota(),
  `wq_cores()` = wq_cores()
)
# A tibble: 1 × 3
  `detectCores()` `cgroup quota` `wq_cores()`
            <int>          <int>        <int>
1               8              8            7

Those two agree on the session this page was rendered in, and that is the uninteresting case. When they disagree you are in a container whose CPU quota is smaller than the machine it sits on, which is the normal case on a hosted workbench and on a Databricks driver, and it is worth checking rather than assuming: the same code moved to a smaller session is where this bites.

Forking past the quota does not queue politely. Each worker is a real process holding real memory, so the job overshoots its memory limit and the session is killed. That arrives as a crash with no R error at all, which makes it the least diagnosable failure in this whole example.

wq_cores() in R/setup.R reads the quota and takes the smaller number. On a two-core quota it returns one, and one is the honest answer: serial, and not a failure.

Here is the shape, on a deliberately trivial unit of work so the mechanism is visible rather than buried:

slow_square <- function(x) {
  Sys.sleep(0.05)
  x^2
}

serial <- system.time(
  map_dbl(1:24, slow_square)
)

forked <- system.time(
  unlist(mclapply(1:24, slow_square, mc.cores = n_cores))
)

tibble(
  run = c("serial", glue("mclapply, {n_cores} cores")),
  seconds = c(serial[["elapsed"]], forked[["elapsed"]])
)
# A tibble: 2 × 2
  run               seconds
  <chr>               <dbl>
1 serial              1.20 
2 mclapply, 7 cores   0.211

Both numbers come from one run in a session of this size, and neither is a benchmark. The point is the ratio being roughly the core count, not the seconds.

Memory goes down, not up

Worth a short section because it removes a worry that would otherwise be carried into the next two stages.

before <- gc(reset = TRUE)

invisible(mclapply(
  1:24,
  \(i) sum(runif(2e5)),
  mc.cores = n_cores
))

peak_mb <- sum(gc()[, "max used"] * c(8, 8)) / 2^20
round(peak_mb, 1)
[1] 18.2

Reading shared data across forked workers costs almost nothing, because the fork is copy-on-write and none of them writes to it. So the arithmetic from Work with a table that will not fit in your session does not transfer here.

What does cost is anything a worker builds for itself. Multiply it by the worker count, because none of it is shared: n workers each constructing a spatial index over the same polygons pay for n indexes. That is the term to watch, and it is the one that goes wrong quietly.

Too few workers is cheap, too many is not

The two directions are not symmetric, which is worth knowing before you tune anything.

Asking for fewer workers than you could have is mild: the job takes longer and nothing else happens. Asking for more is not, and it is not a slowdown either. Past the quota the work does not fit, and a session that is killed for exceeding its memory limit reports no error you can read.

So round down. Leaving a core for the parent, as above, is a habit rather than a rule, but taking the quota rather than the machine count is a rule.

What this decides for the rest of the example

The backend is settled here, and the next two pages lean on it rather than repeating it.

Next

Simulating from a summary.

This page rests on: future and furrr are absent from the cluster runtime bundle while sf, dplyr and base parallel are present on the workers, confirmed by asking the workers directly; forked parallel workers share what the parent already held, because the fork is copy-on-write, but each pays in full for whatever it allocates itself; detectCores() reports the host’s cores rather than the container’s CPU quota, so forking on its answer overcommits and the session is killed without an error message; whether a future multicore plan behaves correctly under a Spark executor is untested.