sdf_unnest_longer

Unnest longer

Description

Expand a struct column or an array column within a Spark dataframe into one or more rows, similar what to tidyr::unnest_longer does to an R dataframe. An index column, if included, will be 1-based if col is an array column.

Usage

sdf_unnest_longer(
  data,
  col,
  values_to = NULL,
  indices_to = NULL,
  include_indices = NULL,
  names_repair = "check_unique",
  ptype = list(),
  transform = list()
)

Arguments

Argument Description
data The Spark dataframe to be unnested
col The struct column to extract components from
values_to Name of column to store vector values. Defaults to col.
indices_to A string giving the name of column which will contain the inner names or position (if not named) of the values. Defaults to col with _id suffix
include_indices Whether to include an index column. An index column will be included by default if col is a struct column. It will also be included if indices_to is not NULL.
names_repair Strategy for fixing duplicate column names (the semantic will be exactly identical to that of .name_repair option in tibble)
ptype Optionally, supply an R data frame prototype for the output. Each column of the unnested result will be casted based on the Spark equivalent of the type of the column with the same name within ptype, e.g., if ptype has a column x of type character, then column x of the unnested result will be casted from its original SQL type to StringType.
transform Optionally, a named list of transformation functions applied

Examples


library(sparklyr)
sc <- spark_connect(master = "local", version = "2.4.0")

# unnesting a struct column
sdf <- copy_to(
  sc,
  tibble::tibble(
    x = 1:3,
    y = list(list(a = 1, b = 2), list(a = 3, b = 4), list(a = 5, b = 6))
  )
)

unnested <- sdf %>% sdf_unnest_longer(y, indices_to = "attr")

# unnesting an array column
sdf <- copy_to(
  sc,
  tibble::tibble(
    x = 1:3,
    y = list(1:10, 1:5, 1:2)
  )
)

unnested <- sdf %>% sdf_unnest_longer(y, indices_to = "array_idx")