Apply an R Function in Spark
Applies an R function to a Spark object (typically, a Spark DataFrame).
spark_apply(x, f, columns = colnames(x), memory = TRUE, group_by = NULL, packages = TRUE, context = NULL, ...)
An object (usually a
A function that transforms a data frame partition into a data frame.
A vector of column names or a named vector of column types for the transformed object. Defaults to the names from the original object and adds indexed column names when not enough columns are specified.
Boolean; should the table be cached into memory?
Column name used to group by data frame partitions.
Boolean to distribute
For clusters using Livy or Yarn cluster mode,
For offline clusters where
Optional object to be serialized and passed back to
Optional arguments; currently unused.