R on Databricks

A working guide for a research scientist whose data lives in Databricks

You write R. Your data is in Databricks. Almost everything published about Spark is written for someone administering a platform, and you are trying to finish an analysis.

This guide is for you, and for nobody else. It assumes you read R, the tidyverse, sf and tidymodels fluently, and it spends its whole explaining budget on the other thing: where your code runs, what has to move before it can run there, and what it costs you.

You were working on your laptop, and now you are working on Databricks

And it is harder. That is not your imagination, and it is worth saying before anything else.

The reason the choice may not be yours is usually one of three:

  • the data is too big to move
  • governance decides where it lives
  • the compute genuinely wins

If none of those hold, and the data fits in your session, work locally. Local R is quicker to set up, easier to debug, and better for your science. Nothing on this site argues otherwise.

When one of them does hold, this guide exists to close as much of the gap as can be closed.

Two shapes of work, and a third

Most of what you will do falls into one of these. They are not a taxonomy to learn, and you do not have to decide which one you are in before reading on. Look for the one that resembles code you have written.

When the table is large, one line decides everything.

readings |>                  # a large table
  inner_join(gauges) |>      #   on Databricks
  filter(!is.na(value)) |>   #   on Databricks
  group_by(station_id) |>    #   on Databricks
  summarise(mean(value)) |>  #   on Databricks
  collect() |>               # <== how much crosses?
  st_as_sf(crs = 27700)      # in R

Everything above collect() runs on Databricks as SQL. Everything below it runs in your own R session, with all of sf and ggplot2 and no translation. The whole question is how big the data is when it crosses. That is Work with a table that will not fit in your session.

When the repeats are the expensive part, almost nothing crosses.

sites <- readings |>         # the same table
  group_by(site_id) |>       #   on Databricks
  summarise(fit_params()) |> #   on Databricks
  collect()                  # <== a small result

exceedances <- sites |>
  split(sites$site_id) |>
  map(simulate_years)        # in R, expensive

Here the crossing is free and the cost is in the last line, repeated. So the question is not how much data moves. It is how many repeats run at once, and whether R is present where they run. That is Run a Monte Carlo simulation.

The third shape is gridded data, and it has no server-side path at all: Databricks’ spatial functions work on vectors, and there is no equivalent for a raster cell. So the only question there is which side of the boundary the pixels sit on. See Work with rasters, using terra.

Where to go

If you have ten minutes and no particular emergency, start with What changes when the data is not on your laptop, under Get started above. It is the one page the rest of the site assumes, and almost every surprise documented here is the same handful of assumptions failing.

If you arrived with a specific problem, go straight to it. These are the Guides.

If you would rather see one analysis carried the whole way through, with the awkward parts left in, that is the worked example.

And if what is stopping you turns out not to be yours to fix, what to ask for, and how to ask is about getting the change made by whoever administers your workspace.

What this guide does not cover

Knowing what you can skip is worth as much as knowing where to start. There is nothing here about streaming, ML pipelines, Python interop, or Spark tuning. If your problem is one of those, this is not the guide, and reading it will not help.

Two things are covered but not settled, and you would make a worse decision without knowing which:

  • Whether spark_apply() works on a Standard access-mode cluster. Settling it needs a cluster that has to be created rather than started, and this guide will not assert it in either direction. It matters because your group cluster is quite likely Standard.
  • How work spread across worker nodes scales. That it spreads at all has been run and observed on a two-worker cluster. Two workers answers whether it distributes and nothing more: it says nothing about shuffle cost, or about behaviour at twenty nodes.

Where anything else on the site is unsettled, the page says so in the prose.

Behind the numbers

Every number here was run and recorded rather than estimated, and none of them is a benchmark.

The examples use public UK environmental data. The data and its licences gives the provenance and the attribution each licence requires. Nothing here is a finding about water: the data is a vehicle for demonstrating technique.