Returns the last index of a Spark DataFrame

R/sdf_interface.R

sdf_last_index

Description

Returns the last index of a Spark DataFrame. The Spark mapPartitionsWithIndex function is used to iterate through the last nonempty partition of the RDD to find the last record.

Usage

sdf_last_index(x, id = "id") 

Arguments

Arguments Description
x A spark_connection, ml_pipeline, or a tbl_spark.
id The name of the index column.