Combines 2 Array Columns

R/dplyr_hof.R

hof_zip_with

Description

Applies an element-wise function to combine elements from 2 array columns (this is essentially a dplyr wrapper for the zip_with(array<T>, array<U>, function<T, U, R>): array<R> built-in function in Spark SQL)

Usage

 
hof_zip_with(x, func, dest_col = NULL, left = NULL, right = NULL, ...) 

Arguments

Arguments Description
x The Spark data frame to process
func Element-wise combining function to be applied
dest_col Column to store the query result (default: the last column of the Spark data frame)
left Any expression evaluating to an array (default: the first column of the Spark data frame)
right Any expression evaluating to an array (default: the second column of the Spark data frame)
Additional params to dplyr::mutate

Examples

 
 
library(sparklyr) 
sc <- spark_connect(master = "local") 
# compute element-wise products of 2 arrays from each row of `left` and `right` 
# and store the resuling array in `res` 
copy_to( 
  sc, 
  tibble::tibble( 
    left = list(1:5, 21:25), 
    right = list(6:10, 16:20), 
    res = c(0, 0) 
  ) 
) %>% 
  hof_zip_with(~ .x * .y) 
#> # Source: spark<?> [?? x 3]
#>   left      right     res      
#>   <list>    <list>    <list>   
#> 1 <dbl [5]> <dbl [5]> <dbl [5]>
#> 2 <dbl [5]> <dbl [5]> <dbl [5]>