Create a Spark Configuration for Livy

R/livy_connection.R

livy_config

Description

Create a Spark Configuration for Livy

Usage

livy_config( 
  config = spark_config(), 
  username = NULL, 
  password = NULL, 
  negotiate = FALSE, 
  custom_headers = list(`X-Requested-By` = "sparklyr"), 
  proxy = NULL, 
  curl_opts = NULL, 
  ... 
) 

Arguments

Arguments	Description
config	Optional base configuration
username	The username to use in the Authorization header
password	The password to use in the Authorization header
negotiate	Whether to use gssnegotiate method or not
custom_headers	List of custom headers to append to http requests. Defaults to list("X-Requested-By" = "sparklyr").
proxy	Either NULL or a proxy specified by httr::use_proxy(). Defaults to NULL.
curl_opts	List of CURL options (e.g., verbose, connecttimeout, dns_cache_timeout, etc, see httr::httr_options() for a list of valid options) – NOTE: these configurations are for libcurl only and separate from HTTP headers or Livy session parameters.
…	additional Livy session parameters

Details

Extends a Spark spark_config() configuration with settings for Livy. For instance, username and password

define the basic authentication settings for a Livy session.

The default value of "custom_headers" is set to list("X-Requested-By" = "sparklyr")

in order to facilitate connection to Livy servers with CSRF protection enabled.

Additional parameters for Livy sessions are:

proxy_user

User to impersonate when starting the session

jars

jars to be used in this session

py_files

Python files to be used in this session

files

files to be used in this session

driver_memory

Amount of memory to use for the driver process

driver_cores

Number of cores to use for the driver process

executor_memory

Amount of memory to use per executor process

executor_cores

Number of cores to use for each executor

num_executors

Number of executors to launch for this session

archives

Archives to be used in this session

queue

The name of the YARN queue to which submitted

name

The name of this session

heartbeat_timeout

Timeout in seconds to which session be orphaned

conf

Spark configuration properties (Map of key=value)

Note that queue is supported only by version 0.4.0 of Livy or newer. If you are using the older one, specify queue via config (e.g. config = spark_config(spark.yarn.queue = "my_queue")).

Value

Named list with configuration data