Read file(s) into a Spark DataFrame using a custom reader

Run a custom R function on Spark workers to ingest data from one or more files into a Spark DataFrame, assuming all files follow the same schema.

spark_read(sc, paths, reader, columns, packages = TRUE, ...)

Arguments

sc

A spark_connection.

paths

A character vector of one or more file URIs (e.g., c("hdfs://localhost:9000/file.txt", "hdfs://localhost:9000/file2.txt"))

reader

A self-contained R function that takes a single file URI as argument and returns the data read from that file as a data frame.

columns

a named list of column names and column types of the resulting data frame (e.g., list(column_1 = "integer", column_2 = "character")), or a list of column names only if column types should be inferred from the data (e.g., list("column_1", "column_2"), or NULL if column types should be inferred and resulting data frame can have arbitrary column names

packages

A list of R packages to distribute to Spark workers

...

Optional arguments; currently unused.

See also

Examples

if (FALSE) { library(sparklyr) sc <- spark_connect( master = "yarn", spark_home = "~/spark/spark-2.4.5-bin-hadoop2.7" ) # This is a contrived example to show reader tasks will be distributed across # all Spark worker nodes spark_read( sc, rep("/dev/null", 10), reader = function(path) system("hostname", intern = TRUE), columns = c(hostname = "string") ) %>% sdf_collect() }