R interface for MLeap

mleap is a sparklyr extension that provides an interface to MLeap, which allows us to take Spark pipelines to production.

Install mleap

mleap can be installed from CRAN via

install.packages("mleap")

or, for the latest development version from GitHub, using

devtools::install_github("rstudio/mleap")

Setup

Once mleap has been installed, we can install the external dependencies using:

library(mleap)
install_mleap()

Another dependency of mleap is Maven. If it is already installed, just point mleap to its location:

options(maven.home = "path/to/maven")

If Maven is not yet installed, which is the most likely case, use the following to install it:

install_maven()

Create an MLeap Bundle

  1. Start Spark session using sparklyr

    library(sparklyr)
    sc <- spark_connect(master = "local", version = "2.2.0")
    mtcars_tbl <- sdf_copy_to(sc, mtcars, overwrite = TRUE)
  2. Create a fit an ML Pipeline

    pipeline <- ml_pipeline(sc) %>%
      ft_binarizer("hp", "big_hp", threshold = 100) %>%
      ft_vector_assembler(c("big_hp", "wt", "qsec"), "features") %>%
      ml_gbt_regressor(label_col = "mpg")
    
    pipeline_model <- ml_fit(pipeline, mtcars_tbl)
  3. A transformed data frame with the appropriate schema is required for exporting the Pipeline model

    transformed_tbl <- ml_transform(pipeline_model, mtcars_tbl)
  4. Export the model using the ml_write_bundle() function from mleap

    model_path <- file.path(tempdir(), "mtcars_model.zip")
    ml_write_bundle(pipeline_model, transformed_tbl, model_path)
    ## Model successfully exported.
  5. Close Spark session

    spark_disconnect(sc)

At this point, we can share mtcars_model.zip with the deployment/implementation engineers, and they would be able to embed the model in another application. See the MLeap docs for details.

Test the mleap bundle

The mleap package also provides R functions for testing that the saved models behave as expected. Here we load the previously saved model:

model <- mleap_load_bundle(model_path)
model
## MLeap Transformer
## <db23a9f1-7b3d-4d27-9eb0-8675125ab3a5> 
##   Name: pipeline_fe6b8cb0028f 
##   Format: json 
##   MLeap Version: 0.10.0-SNAPSHOT

To retrieve the schema associated with the model use the mleap_model_schema() function

mleap_model_schema(model)
## # A tibble: 6 x 4
##   name       type   nullable dimension
##   <chr>      <chr>  <lgl>    <chr>    
## 1 qsec       double TRUE     <NA>     
## 2 hp         double FALSE    <NA>     
## 3 wt         double TRUE     <NA>     
## 4 big_hp     double FALSE    <NA>     
## 5 features   double TRUE     (3)      
## 6 prediction double FALSE    <NA>

Then, we create a new data frame to be scored, and make predictions using the model:

newdata <- tibble::tribble(
  ~qsec, ~hp, ~wt,
  16.2,  101, 2.68,
  18.1,  99,  3.08
)

# Transform the data frame
transformed_df <- mleap_transform(model, newdata)
dplyr::glimpse(transformed_df)
## Observations: 2
## Variables: 6
## $ qsec       <dbl> 16.2, 18.1
## $ hp         <dbl> 101, 99
## $ wt         <dbl> 2.68, 3.08
## $ big_hp     <dbl> 1, 0
## $ features   <list> [[[1, 2.68, 16.2], [3]], [[0, 3.08, 18.1], [3]]]
## $ prediction <dbl> 21.06529, 22.36667