The future.batchtools package is a backend for the future
framework, also known as the Futureverse. The Futureverse makes it
easy to parallelize existing R code - often with only a minor change
of code. Most users interact with it via higher-level packages such
as futurize, future.apply, furrr, and
doFuture, and you choose where and how the code runs by
setting a future backend with plan(). This package adds backends
that run futures as jobs on a high-performance compute (HPC) cluster,
via job schedulers such as Slurm, SGE, LSF, and TORQUE/PBS, using the
batchtools package.
library(futurize)
plan(future.batchtools::batchtools_slurm) # parallelize via Slurm jobs
# A slow function
slow_fcn <- function(x) {
Sys.sleep(1.0)
x^2
}
# Example data
xs <- 1:10
# Sequential and parallel base R apply
ys <- lapply(xs, slow_fcn)
ys <- lapply(xs, slow_fcn) |> futurize()
# Sequential and parallel purrr map
library(purrr)
ys <- xs |> map(slow_fcn)
ys <- xs |> map(slow_fcn) |> futurize()
# Sequential and parallel foreach calls
library(foreach)
ys <- foreach(x = xs) %do% slow_fcn(x)
ys <- foreach(x = xs) %do% slow_fcn(x) |> futurize()The only thing that differs from using any other future backend is
the plan() call. The code does not have to change, it gives
identical results, and it behaves exactly the same.
| Backend | Description | Alternative in future package |
|---|---|---|
batchtools_lsf |
Futures are evaluated via a Load Sharing Facility (LSF) job scheduler | N/A |
batchtools_openlava |
Futures are evaluated via an OpenLava job scheduler | N/A |
batchtools_sge |
Futures are evaluated via a Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE) job scheduler | N/A |
batchtools_slurm |
Futures are evaluated via a Slurm job scheduler | N/A |
batchtools_torque |
Futures are evaluated via a TORQUE / PBS job scheduler | N/A |
batchtools_custom |
Futures are evaluated via a custom batchtools configuration R script or via a set of cluster functions | N/A |
batchtools_multicore |
parallel evaluation by forking the current R process | plan(multicore) |
batchtools_local |
sequential evaluation in a separate R process (on current machine) | plan(cluster, workers = I(1)) |
Each future is submitted as a job to the job scheduler. You can
request the resources each job needs via argument resources, e.g.
# Limit runtime to 10 minutes and memory to 400 MiB per future, and
# load environment module 'r/4.5' in the job script
plan(future.batchtools::batchtools_slurm, resources = list(
time = "00:10:00", mem = "400M", modules = c("r/4.5")
))-
You can harness the compute power of an HPC cluster from within R, without having to write job scripts or change any code.
-
You can specify the resources that the jobs need, e.g. runtime, memory, CPU cores, and environment modules, via
plan(). -
All of the common job schedulers are supported, and others can be supported via a custom batchtools configuration.
-
Each future is a separate job, which means it has to wait in the job queue before it starts. This makes the batchtools backends best suited for futures that take minutes or longer to run, rather than many small futures.
-
The batchtools registry folder, which is
.future/in the current working directory by default, must be on a file system that is shared between the machine submitting the jobs and the compute nodes.
R package future.batchtools is available on CRAN and can be installed in R as:
install.packages("future.batchtools")To install the pre-release version, use:
install.packages("future.batchtools", repos = c("https://futureverse-devel.r-universe.dev", "https://cloud.r-project.org"))