hof_zip_with
Combines 2 Array Columns
Description
Applies an element-wise function to combine elements from 2 array columns (this is essentially a dplyr wrapper for the zip_with(array<T>, array<U>, function<T, U, R>): array<R>
built-in function in Spark SQL)
Usage
hof_zip_with(x, func, dest_col = NULL, left = NULL, right = NULL, ...)
Arguments
Argument | Description |
---|---|
x | The Spark data frame to process |
func | Element-wise combining function to be applied |
dest_col | Column to store the query result (default: the last column of the Spark data frame) |
left | Any expression evaluating to an array (default: the first column of the Spark data frame) |
right | Any expression evaluating to an array (default: the second column of the Spark data frame) |
… | Additional params to dplyr::mutate |
Examples
library(sparklyr)
<- spark_connect(master = "local")
sc # compute element-wise products of 2 arrays from each row of `left` and `right`
# and store the resuling array in `res`
copy_to(
sc,::tibble(
tibbleleft = list(1:5, 21:25),
right = list(6:10, 16:20),
res = c(0, 0)
)%>%
) hof_zip_with(~ .x * .y)