Spark Operations |
|
---|---|
Read Spark Configuration |
|
|
Manage Spark Connections |
|
Find a given Spark installation by version. |
View Entries in the Spark Log |
|
Open the Spark web interface |
|
Check whether the connection is open |
|
A Shiny app that can be used to construct a |
|
Runtime configuration interface for the Spark Session |
|
Set/Get Spark checkpoint directory |
|
Generate a Table Name from Expression |
|
Get the Spark Version Associated with a Spark Installation |
|
Retrieves a dataframe available Spark versions that van be installed. |
|
Kubernetes Configuration |
|
Retrieve Available Settings |
|
Find Spark Connection |
|
Fallback to Spark Dependency |
|
Create Spark Extension |
|
Reads from a Spark Table into a Spark DataFrame. |
|
Read libsvm file into a Spark DataFrame. |
|
list all sparklyr-*.jar files that have been built |
|
Creates Spark Configuration |
|
Retrieve the Spark Connection Associated with an R Object |
|
Spark Data |
|
Read file(s) into a Spark DataFrame using a custom reader |
|
Read Apache Avro data into a Spark DataFrame. |
|
Read a CSV file into a Spark DataFrame |
|
Read from Delta Lake into a Spark DataFrame. |
|
Read from JDBC connection into a Spark DataFrame. |
|
Read a JSON file into a Spark DataFrame |
|
Read a Parquet file into a Spark DataFrame |
|
Read from a generic source into a Spark DataFrame. |
|
Reads from a Spark Table into a Spark DataFrame. |
|
Read a ORC file into a Spark DataFrame |
|
Read a Text file into a Spark DataFrame |
|
Saves a Spark DataFrame as a Spark table |
|
Write Spark DataFrame to file using a custom writer |
|
Serialize a Spark DataFrame into Apache Avro format |
|
Write a Spark DataFrame to a ORC file |
|
Write a Spark DataFrame to a Text file |
|
Write a Spark DataFrame to a CSV |
|
Writes a Spark DataFrame into Delta Lake |
|
Writes a Spark DataFrame into a JDBC table |
|
Write a Spark DataFrame to a JSON file |
|
Write a Spark DataFrame to a Parquet file |
|
Writes a Spark DataFrame into a generic source |
|
Writes a Spark DataFrame into a Spark table |
|
Spark Tables |
|
Show database list |
|
Cache a Spark Table |
|
Use specific database |
|
Uncache a Spark Table |
|
Spark DataFrames |
|
Copy an R Data Frame to Spark |
|
Create DataFrame for along Object |
|
Bind multiple Spark DataFrames by row and column |
|
Broadcast hint |
|
Checkpoint a Spark DataFrame |
|
Coalesces a Spark DataFrame |
|
Copy an Object into Spark |
|
Remove duplicates from a Spark DataFrame |
|
Convert column(s) from avro format |
|
Create DataFrame for Length |
|
Gets number of partitions of a Spark DataFrame |
|
Partition a Spark Dataframe |
|
Pivot a Spark DataFrame |
|
|
Spark ML -- Transform, fit, and predict methods (sdf_ interface) |
Read a Column from a Spark DataFrame |
|
Register a Spark DataFrame |
|
Repartition a Spark DataFrame |
|
Model Residuals |
|
Randomly Sample Rows from a Spark DataFrame |
|
Separate a Vector Column into Scalar Columns |
|
Create DataFrame for Range |
|
Sort a Spark DataFrame |
|
Convert column(s) to avro format |
|
Add a Unique ID Column to a Spark DataFrame |
|
Collect a Spark DataFrame into R. |
|
Cross Tabulation |
|
Debug Info for Spark DataFrame |
|
Compute summary statistics for columns of a data frame |
|
Support for Dimension Operations |
|
Spark DataFrame is Streaming |
|
Returns the last index of a Spark DataFrame |
|
|
Save / Load a Spark DataFrame |
Persist a Spark DataFrame |
|
Project features onto principal components |
|
Compute (Approximate) Quantiles with a Spark DataFrame |
|
Read the Schema of a Spark DataFrame |
|
Spark DataFrame from SQL |
|
Add a Sequential ID Column to a Spark DataFrame |
|
|
Join Spark tbls. |
Separate |
|
Unite |
|
Nest |
|
Unnest |
|
Pivot wider |
|
Pivot longer |
|
Fill |
|
Left join |
|
Right join |
|
Inner join |
|
Full join |
|
Apply Aggregate Function to Array Column |
|
Sorts array using a custom comparator |
|
Determine Whether Some Element Exists in an Array Column |
|
Filter Array Column |
|
Checks whether all elements in an array satisfy a predicate |
|
Filters a map |
|
Merges two maps into one |
|
Transform Array Column |
|
Transforms keys of a map |
|
Transforms values of a map |
|
Combines 2 Array Columns |
|
Perform Weighted Random Sampling on a Spark DataFrame |
|
transform a subset of column(s) in a Spark Dataframe |
|
Spark Machine Learning |
|
|
Spark ML -- Decision Trees |
Spark ML -- Generalized Linear Regression |
|
|
Spark ML -- Gradient Boosted Trees |
Spark ML -- K-Means Clustering |
|
|
Spark ML -- Latent Dirichlet Allocation |
Spark ML -- Linear Regression |
|
Spark ML -- Logistic Regression |
|
Extracts data associated with a Spark ML model |
|
|
Spark ML -- Multilayer Perceptron |
Spark ML -- Naive-Bayes |
|
Spark ML -- OneVsRest |
|
Feature Transformation -- PCA (Estimator) |
|
|
Spark ML -- Random Forest |
Spark ML -- Survival Regression |
|
Add a Stage to a Pipeline |
|
Spark ML -- ALS |
|
Utility functions for LSH models |
|
Frequent Pattern Mining -- FPGrowth |
|
|
Spark ML - Evaluators |
Spark ML -- Bisecting K-Means Clustering |
|
Wrap a Spark ML JVM object |
|
Chi-square hypothesis testing for categorical data. |
|
Spark ML - Clustering Evaluator |
|
|
Constructors for `ml_model` Objects |
Compute correlation matrix |
|
|
Spark ML -- Tuning |
Default stop words |
|
Evaluate the Model on a Validation Set |
|
Spark ML - Feature Importance for Tree Models |
|
Feature Transformation -- Word2Vec (Estimator) |
|
|
Spark ML -- Transform, fit, and predict methods (ml_ interface) |
Spark ML -- Gaussian Mixture clustering. |
|
Spark ML -- ML Params |
|
Spark ML -- Isotonic Regression |
|
Feature Transformation -- StringIndexer (Estimator) |
|
Spark ML -- LinearSVC |
|
Spark ML -- Model Persistence |
|
Spark ML -- Pipelines |
|
Spark ML -- Pipeline stage extraction |
|
Standardize Formula Input for `ml_model` |
|
Spark ML -- Extraction of summary metrics |
|
Spark ML -- UID |
|
Feature Transformation -- CountVectorizer (Estimator) |
|
Spark Feature Transformers |
|
Feature Transformation -- Binarizer (Transformer) |
|
Feature Transformation -- Bucketizer (Transformer) |
|
Feature Transformation -- CountVectorizer (Estimator) |
|
Feature Transformation -- Discrete Cosine Transform (DCT) (Transformer) |
|
Feature Transformation -- ElementwiseProduct (Transformer) |
|
Feature Transformation -- IndexToString (Transformer) |
|
Feature Transformation -- OneHotEncoder (Transformer) |
|
Feature Transformation -- QuantileDiscretizer (Estimator) |
|
Feature Transformation -- SQLTransformer |
|
Feature Transformation -- StringIndexer (Estimator) |
|
Feature Transformation -- VectorAssembler (Transformer) |
|
Feature Transformation -- Tokenizer (Transformer) |
|
Feature Transformation -- RegexTokenizer (Transformer) |
|
Feature Transformation -- LSH (Estimator) |
|
Feature Transformation -- ChiSqSelector (Estimator) |
|
Feature Transformation -- FeatureHasher (Transformer) |
|
Feature Transformation -- HashingTF (Transformer) |
|
Feature Transformation -- IDF (Estimator) |
|
Feature Transformation -- Imputer (Estimator) |
|
Feature Transformation -- Interaction (Transformer) |
|
Feature Transformation -- MaxAbsScaler (Estimator) |
|
Feature Transformation -- MinMaxScaler (Estimator) |
|
Feature Transformation -- NGram (Transformer) |
|
Feature Transformation -- Normalizer (Transformer) |
|
Feature Transformation -- OneHotEncoderEstimator (Estimator) |
|
Feature Transformation -- PCA (Estimator) |
|
Feature Transformation -- PolynomialExpansion (Transformer) |
|
Feature Transformation -- RFormula (Estimator) |
|
Feature Transformation -- StandardScaler (Estimator) |
|
Feature Transformation -- StopWordsRemover (Transformer) |
|
Feature Transformation -- VectorIndexer (Estimator) |
|
Feature Transformation -- VectorSlicer (Transformer) |
|
Feature Transformation -- Word2Vec (Estimator) |
|
Feature Transformation -- RobustScaler (Estimator) |
|
Spark Machine Learning Utilities |
|
|
Spark ML - Evaluators |
Spark ML - Feature Importance for Tree Models |
|
|
Tidying methods for Spark ML ALS |
|
Tidying methods for Spark ML linear models |
|
Tidying methods for Spark ML Isotonic Regression |
|
Tidying methods for Spark ML LDA models |
|
Tidying methods for Spark ML linear svc |
|
Tidying methods for Spark ML Logistic Regression |
|
Tidying methods for Spark ML MLP |
|
Tidying methods for Spark ML Naive Bayes |
|
Tidying methods for Spark ML Principal Component Analysis |
|
Tidying methods for Spark ML Survival Regression |
|
Tidying methods for Spark ML tree models |
|
Tidying methods for Spark ML unsupervised models |
Extensions |
|
Compile Scala sources into a Java Archive (jar) |
|
Read configuration values for a connection |
|
Downloads default Scala Compilers |
|
Discover the Scala Compiler |
|
|
Access the Spark API |
Runtime configuration interface for Hive |
|
Invoke a Method on a JVM Object |
|
Register a Package that Implements a Spark Extension |
|
Define a Spark Compilation Specification |
|
Default Compilation Specification for Spark Extensions |
|
Runtime configuration interface for the Spark Context. |
|
Retrieve a Spark DataFrame |
|
Define a Spark dependency |
|
Set the SPARK_HOME environment variable |
|
Retrieve a Spark JVM Object Reference |
|
Get the Spark Version Associated with a Spark Connection |
|
Distributed Computing |
|
Apply an R Function in Spark |
|
Create Bundle for Spark Apply |
|
Log Writer for Spark Apply |
|
Register a Parallel Backend |
|
Livy |
|
|
Install Livy |
Create a Spark Configuration for Livy |
|
Start Livy |
|
Streaming |
|
Find Stream |
|
Generate Test Stream |
|
Spark Stream's Identifier |
|
Spark Stream's Name |
|
Read CSV Stream |
|
Read JSON Stream |
|
Read Delta Stream |
|
Read Kafka Stream |
|
Read ORC Stream |
|
Read Parquet Stream |
|
Read Socket Stream |
|
Read Text Stream |
|
Render Stream |
|
Stream Statistics |
|
Stops a Spark Stream |
|
Spark Stream Continuous Trigger |
|
Spark Stream Interval Trigger |
|
View Stream |
|
Watermark Stream |
|
Write Console Stream |
|
Write CSV Stream |
|
Write Delta Stream |
|
Write JSON Stream |
|
Write Kafka Stream |
|
Write Memory Stream |
|
Write a ORC Stream |
|
Write Parquet Stream |
|
Write Text Stream |
|
Reactive spark reader |