Merges two maps into a single map by applying the function specified to pairs of values with the same key (this is essentially a dplyr wrapper to the `map_zip_with(map1, map2, func)` higher- order function, which is supported since Spark 3.0)

hof_map_zip_with(x, func, dest_col = NULL, map1 = NULL, map2 = NULL, ...)

Arguments

x

The Spark data frame to be processed

func

The function to apply (it should take (key, value1, value2) as arguments, where (key, value1) is a key-value pair present in map1, (key, value2) is a key-value pair present in map2, and return a transformed value associated with key in the resulting map

dest_col

Column to store the query result (default: the last column of the Spark data frame)

map1

The first map being merged, could be any SQL expression evaluating to a map (default: the first column of the Spark data frame)

map2

The second map being merged, could be any SQL expression evaluating to a map (default: the second column of the Spark data frame)

...

Additional params to dplyr::mutate

Examples

if (FALSE) { library(sparklyr) sc <- spark_connect(master = "local", version = "3.0.0") # create a Spark dataframe with 2 columns of type MAP<STRING, INT> two_maps_tbl <- sdf_copy_to( sc, tibble::tibble( m1 = c("{\"1\":2,\"3\":4,\"5\":6}", "{\"2\":1,\"4\":3,\"6\":5}"), m2 = c("{\"1\":1,\"3\":3,\"5\":5}", "{\"2\":2,\"4\":4,\"6\":6}") ), overwrite = TRUE ) %>% dplyr::mutate(m1 = from_json(m1, "MAP<STRING, INT>"), m2 = from_json(m2, "MAP<STRING, INT>")) # create a 3rd column containing MAP<STRING, INT> values derived from the # first 2 columns transformed_two_maps_tbl <- two_maps_tbl %>% hof_map_zip_with( func = .(k, v1, v2) %->% (CONCAT(k, "_", v1, "_", v2)), dest_col = m3 ) }