Introduction to xform_function

Source: vignettes/xform_function.Rmd

Introduction

This vignette provides examples of how to use the xform_function transformation to create new data features for PMML models.

Given a xform_wrap object and a transformation expression, xform_function calculates data for a new feature and creates a new xform_wrap object. When PMML is produced with pmml::pmml(), the transformation is inserted into the LocalTransformations node as a DerivedField.

Multiple data fields and functions can be combined to produce a new feature.

The code below uses knitr::kable() to make tables more readable.

Single numeric input

Using the iris dataset as an example, let’s construct a new feature by transforming one variable. Load the dataset and show the first few lines:

data(iris)
kable(head(iris,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
5.1 3.5 1.4 0.2 setosa
4.9 3.0 1.4 0.2 setosa
4.7 3.2 1.3 0.2 setosa

Create the iris_box object with xform_wrap:

iris_box <- xform_wrap(iris)

iris_box contains the data and transform information that will be used to produce PMML later. The original data is in iris_box$data. Any new features created with a transformation are added as columns to this data frame.

kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
5.1 3.5 1.4 0.2 setosa
4.9 3.0 1.4 0.2 setosa
4.7 3.2 1.3 0.2 setosa

Transform and field information is in iris_box$field_data. The field_data data frame contains information on every field in the dataset, as well as every transform used. The xform_function column contains expressions used in the xform_function transform.

kable(iris_box$field_data)
type dataType orig_field_name sampleMin sampleMax xformedMin xformedMax centers scales fieldsMap transform default missingValue xform_function
Sepal.Length original numeric NA NA NA NA NA NA NA NA NA NA NA NA
Sepal.Width original numeric NA NA NA NA NA NA NA NA NA NA NA NA
Petal.Length original numeric NA NA NA NA NA NA NA NA NA NA NA NA
Petal.Width original numeric NA NA NA NA NA NA NA NA NA NA NA NA
Species original factor NA NA NA NA NA NA NA NA NA NA NA NA

Now add a new feature, Sepal.Length.Sqrt, using xform_function:

iris_box <- xform_function(iris_box,orig_field_name="Sepal.Length",
                         new_field_name="Sepal.Length.Sqrt",
                         expression="sqrt(Sepal.Length)")

The new feature is calculated and added as a column to the iris_box$data data frame:

kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species Sepal.Length.Sqrt
5.1 3.5 1.4 0.2 setosa 2.258318
4.9 3.0 1.4 0.2 setosa 2.213594
4.7 3.2 1.3 0.2 setosa 2.167948

iris_box$field_data now contains a new row with the transformation expression:

kable(iris_box$field_data[6,c(1:3,14)])
type dataType orig_field_name xform_function
Sepal.Length.Sqrt derived numeric Sepal.Length sqrt(Sepal.Length)

Construct a linear model for Petal.Width using this new feature, and convert it to PMML:

fit <- lm(Petal.Width ~ Sepal.Length.Sqrt, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)

Since the model predicts Petal.Width using a variable based on Sepal.Length, the PMML will contain these two fields in the DataDictionary and MiningSchema:

fit_pmml[[2]] #Data Dictionary node
#> <DataDictionary numberOfFields="2">
#>  <DataField name="Petal.Width" optype="continuous" dataType="double"/>
#>  <DataField name="Sepal.Length" optype="continuous" dataType="double"/>
#> </DataDictionary>
fit_pmml[[3]][[1]] #Mining Schema node
#> <MiningSchema>
#>  <MiningField name="Petal.Width" usageType="predicted" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Sepal.Length" usageType="active" invalidValueTreatment="returnInvalid"/>
#> </MiningSchema>

The LocalTransformations node contains Sepal.Length.Sqrt as a derived field:

fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Sepal.Length.Sqrt" dataType="double" optype="continuous">
#> <Apply function="sqrt">
#>   <FieldRef field="Sepal.Length"/>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

Single categorical input

xform_function can also operate on categorical data. In this example, let’s create a numeric feature that equals 1 when Species is setosa, and 0 otherwise:

iris_box <- xform_wrap(iris)
iris_box <- xform_function(iris_box,orig_field_name="Species",
                         new_field_name="Species.Setosa",
                         expression="if (Species == 'setosa') {1} else {0}")
kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species Species.Setosa
5.1 3.5 1.4 0.2 setosa 1
4.9 3.0 1.4 0.2 setosa 1
4.7 3.2 1.3 0.2 setosa 1

Create a linear model and check the LocalTransformations node:

fit <- lm(Petal.Width ~ Species.Setosa, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)
fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Species.Setosa" dataType="double" optype="continuous">
#> <Apply function="if">
#>   <Apply function="equal">
#>     <FieldRef field="Species"/>
#>     <Constant dataType="string">setosa</Constant>
#>   </Apply>
#>   <Constant dataType="double">1</Constant>
#>   <Constant dataType="double">0</Constant>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

Multiple input fields

Several fields can be combined to create new features. Let’s make a new field from the ratio of sepal and petal lengths:

iris_box <- xform_wrap(iris)
iris_box <- xform_function(iris_box,orig_field_name="Sepal.Length,Petal.Length",
                         new_field_name="Length.Ratio",
                         expression="Sepal.Length / Petal.Length")

As before, the new field is added as a column to the iris_box$data data frame:

kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species Length.Ratio
5.1 3.5 1.4 0.2 setosa 3.642857
4.9 3.0 1.4 0.2 setosa 3.500000
4.7 3.2 1.3 0.2 setosa 3.615385

Fit a linear model using this new feature, and convert it to pmml:

fit <- lm(Petal.Width ~ Length.Ratio, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)

The pmml will contain Sepal.Length and Petal.Length in the DataDictionary and MiningSchema:

fit_pmml[[2]] #Data Dictionary node
#> <DataDictionary numberOfFields="3">
#>  <DataField name="Petal.Width" optype="continuous" dataType="double"/>
#>  <DataField name="Sepal.Length" optype="continuous" dataType="double"/>
#>  <DataField name="Petal.Length" optype="continuous" dataType="double"/>
#> </DataDictionary>
fit_pmml[[3]][[1]] #Mining Schema node
#> <MiningSchema>
#>  <MiningField name="Petal.Width" usageType="predicted" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Sepal.Length" usageType="active" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Petal.Length" usageType="active" invalidValueTreatment="returnInvalid"/>
#> </MiningSchema>

The Local.Transformations node contains Length.Ratio as a derived field:

fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Length.Ratio" dataType="double" optype="continuous">
#> <Apply function="/">
#>   <FieldRef field="Sepal.Length"/>
#>   <FieldRef field="Petal.Length"/>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

Using a previously derived feature

It is possible to pass a feature derived with xform_function to another xform_function call. To do this, the second call to xform_function must use the original data field names (instead of the derived field) in the orig_field_name argument.

iris_box <- xform_wrap(iris)
iris_box <- xform_function(iris_box,orig_field_name="Sepal.Length,Petal.Length",
                         new_field_name="Length.Ratio",
                         expression="Sepal.Length / Petal.Length")

iris_box <- xform_function(iris_box,orig_field_name="Sepal.Length,Petal.Length,Sepal.Width",
                         new_field_name="Length.R.Times.S.Width",
                         expression="Length.Ratio * Sepal.Width")
kable(iris_box$field_data[6:7,c(1:3,14)])
type dataType orig_field_name xform_function
Length.Ratio derived numeric Sepal.Length,Petal.Length Sepal.Length / Petal.Length
Length.R.Times.S.Width derived numeric Sepal.Length,Petal.Length,Sepal.Width Length.Ratio * Sepal.Width 
fit <- lm(Petal.Width ~ Length.R.Times.S.Width, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)

The pmml will contain Sepal.Length, Petal.Length, and Sepal.Width in the DataDictionary and MiningSchema:

fit_pmml[[2]] #Data Dictionary node
#> <DataDictionary numberOfFields="4">
#>  <DataField name="Petal.Width" optype="continuous" dataType="double"/>
#>  <DataField name="Sepal.Length" optype="continuous" dataType="double"/>
#>  <DataField name="Petal.Length" optype="continuous" dataType="double"/>
#>  <DataField name="Sepal.Width" optype="continuous" dataType="double"/>
#> </DataDictionary>
fit_pmml[[3]][[1]] #Mining Schema node
#> <MiningSchema>
#>  <MiningField name="Petal.Width" usageType="predicted" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Sepal.Length" usageType="active" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Petal.Length" usageType="active" invalidValueTreatment="returnInvalid"/>
#>  <MiningField name="Sepal.Width" usageType="active" invalidValueTreatment="returnInvalid"/>
#> </MiningSchema>

The Local.Transformations node contains Length.Ratio and Length.R.Times.S.Width as derived fields:

fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Length.Ratio" dataType="double" optype="continuous">
#> <Apply function="/">
#>   <FieldRef field="Sepal.Length"/>
#>   <FieldRef field="Petal.Length"/>
#> </Apply> 
#>  </DerivedField>
#>  <DerivedField name="Length.R.Times.S.Width" dataType="double" optype="continuous">
#> <Apply function="*">
#>   <FieldRef field="Length.Ratio"/>
#>   <FieldRef field="Sepal.Width"/>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

Factor output

The resulting field can be numeric or factor. Note that factors are exported with dataType = "string" and optype = "categorical" in PMML. The following code creates a factor with 3 levels from Sepal.Length:

iris_box <- xform_wrap(iris)

iris_box <- xform_function(wrap_object = iris_box,
                              orig_field_name = "Sepal.Length",
                              new_field_name = "SL_factor",
                              new_field_data_type = "factor",
                              expression = "if(Sepal.Length<5.1) {'level_A'} else if (Sepal.Length>6.6) {'level_B'} else {'level_C'}")

kable(head(iris_box$data, 3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species SL_factor
5.1 3.5 1.4 0.2 setosa level_C
4.9 3.0 1.4 0.2 setosa level_A
4.7 3.2 1.3 0.2 setosa level_A

The feature can then be used to create a model as usual:

fit <- lm(Petal.Width ~ SL_factor, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)

PMML functions supported by xform_function

The following R functions and operators are directly supported by xform_function. Their PMML equivalents are listed in the second column:

R PMML
+ +
- -
/ /
* *
^ pow
< lessThan
<= lessOrEqual
> greaterThan
>= greaterOrEqual
&& and
& and
| or
|| or
== equal
!= notEqual
! not
ceiling ceil
prod product
log ln

For these functions, no extra code is required for translation.

The R function prod can be used as long as only numeric arguments are specified. That is, prod can take an na.rm argument, but specifying this in xform_function directly will not produce PMML equivalent to the R expression.

Similarly, the R function log can be used directly as long as the second argument (the base) is not specified.

PMML functions not supported by xform_function

There are built-in functions defined in PMML that cannot be directly translated to PMML using xform_function as described above.

In this case, an error will be thrown when R tries to calculate a new feature using the function passed to xform_function, but does not see that function in the environment.

It is still possible to make xform_function work, but the PMML function must be defined in the R environment first.

Let’s use isIn, a PMML function, as an example. The function returns a boolean indicating whether the first argument is contained in a list of values. Detailed specification for this function is available on this DMG page.

One way to implement this in R is by using %in%, with the list of values being represented by ...:

isIn <- function(x, ...) {
  dots <- c(...)
  if (x %in% dots) {
    return(TRUE)
  } else {
    return(FALSE)
  }
}

isIn(1,2,1,4)
#> [1] TRUE

This function can now be passed to xform_function. The following code creates a feature that indicates whether Species is either setosa or versicolor:

iris_box <- xform_wrap(iris)
iris_box <- xform_function(iris_box,orig_field_name="Species",
                         new_field_name="Species.Setosa.or.Versicolor",
                         expression="isIn(Species,'setosa','versicolor')")

The data data frame now contains the new feature:

kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species Species.Setosa.or.Versicolor
5.1 3.5 1.4 0.2 setosa 1
4.9 3.0 1.4 0.2 setosa 1
4.7 3.2 1.3 0.2 setosa 1

Create a linear model and view the corresponding PMML for the function:

fit <- lm(Petal.Width ~ Species.Setosa.or.Versicolor, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)
fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Species.Setosa.or.Versicolor" dataType="double" optype="continuous">
#> <Apply function="isIn">
#>   <FieldRef field="Species"/>
#>   <Constant dataType="string">setosa</Constant>
#>   <Constant dataType="string">versicolor</Constant>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

PMML function not supported by xform_function - another example

As another example, let’s use R’s mean function to create a new feature. PMML has a built-in avg, so we will define an R function with this name.

avg <- function(...) {
  dots <- c(...)
  return(mean(dots))
}

Now use this function to take an average of several other features and combine with another field:

iris_box <- xform_wrap(iris)
iris_box <- xform_function(iris_box,orig_field_name="Sepal.Length,Petal.Length,Sepal.Width",
                         new_field_name="Length.Average.Ratio",
                         expression="avg(Sepal.Length,Petal.Length)/Sepal.Width")

The data data frame now contains the new feature:

kable(head(iris_box$data,3))
Sepal.Length Sepal.Width Petal.Length Petal.Width Species Length.Average.Ratio
5.1 3.5 1.4 0.2 setosa 0.9285714
4.9 3.0 1.4 0.2 setosa 1.0500000
4.7 3.2 1.3 0.2 setosa 0.9375000

Create a simple linear model and view the corresponding PMML for the function:

fit <- lm(Petal.Width ~ Length.Average.Ratio, data=iris_box$data)
fit_pmml <- pmml(fit, transform=iris_box)
fit_pmml[[3]][[3]]
#> <LocalTransformations>
#>  <DerivedField name="Length.Average.Ratio" dataType="double" optype="continuous">
#> <Apply function="/">
#>   <Apply function="avg">
#>     <FieldRef field="Sepal.Length"/>
#>     <FieldRef field="Petal.Length"/>
#>   </Apply>
#>   <FieldRef field="Sepal.Width"/>
#> </Apply> 
#>  </DerivedField>
#> </LocalTransformations>

In the PMML, avg will be recognized as a valid function.

PMML for arbitrary functions

The function function_to_pmml (part of the pmml package) makes it possible to convert an R expression into PMML directly, without creating a model or calculating values.

As long as the expression passed to the function is a valid R expression (e.g., no unbalanced parentheses), it can contain arbitrary function names not defined in R. Variables in the expression passed to xform_function are always assumed to be field names, and not substituted. That is, even if x has a value in the R environment, the resulting expression will still use x.

function_to_pmml("1 + 2")
#> <Apply function="+">
#>   <Constant dataType="double">1</Constant>
#>   <Constant dataType="double">2</Constant>
#> </Apply>

x <- 3
function_to_pmml("foo(bar(x * y))")
#> <Apply function="foo">
#>   <Apply function="bar">
#>     <Apply function="*">
#>       <FieldRef field="x"/>
#>       <FieldRef field="y"/>
#>     </Apply>
#>   </Apply>
#> </Apply>

More notes on functions

There are several limitations to parsing expressions in xform_function.

Each transformation operates on one data row at a time. For example, it is not possible to compute the mean of an entire feature column in xform_function.

An expression such as foo(x) is treated as a function foo with argument x. Consequently, passing in an R vector c(1,2,3) will produce PMML where c is a function and 1,2,3 are the arguments:

function_to_pmml("c(1,2,3)")
#> <Apply function="c">
#>   <Constant dataType="double">1</Constant>
#>   <Constant dataType="double">2</Constant>
#>   <Constant dataType="double">3</Constant>
#> </Apply>

We can also see what happens when passing an na.rm argument to prod, as mentioned in an above example:

function_to_pmml("prod(1,2,na.rm=FALSE)") #produces incorrect PMML
#> <Apply function="product">
#>   <Constant dataType="double">1</Constant>
#>   <Constant dataType="double">2</Constant>
#>   <Constant dataType="boolean">FALSE</Constant>
#> </Apply>
function_to_pmml("prod(1,2)") #produces correct PMML
#> <Apply function="product">
#>   <Constant dataType="double">1</Constant>
#>   <Constant dataType="double">2</Constant>
#> </Apply>

Additionally, passing in a vector to prod produces incorrect PMML:

prod(c(1,2,3))
#> [1] 6
function_to_pmml("prod(c(1,2,3))")
#> <Apply function="product">
#>   <Apply function="c">
#>     <Constant dataType="double">1</Constant>
#>     <Constant dataType="double">2</Constant>
#>     <Constant dataType="double">3</Constant>
#>   </Apply>
#> </Apply>

More examples of functions

The following are additional examples of pmml produced from R expressions.

Extra parentheses:

function_to_pmml("pmmlT(((1+2))*(x))")
#> <Apply function="pmmlT">
#>   <Apply function="*">
#>     <Apply function="+">
#>       <Constant dataType="double">1</Constant>
#>       <Constant dataType="double">2</Constant>
#>     </Apply>
#>     <FieldRef field="x"/>
#>   </Apply>
#> </Apply>

If-else expressions:

function_to_pmml("if(a<2) {x+3} else if (a>4) {4} else {5}")
#> <Apply function="if">
#>   <Apply function="lessThan">
#>     <FieldRef field="a"/>
#>     <Constant dataType="double">2</Constant>
#>   </Apply>
#>   <Apply function="+">
#>     <FieldRef field="x"/>
#>     <Constant dataType="double">3</Constant>
#>   </Apply>
#>   <Apply function="if">
#>     <Apply function="greaterThan">
#>       <FieldRef field="a"/>
#>       <Constant dataType="double">4</Constant>
#>     </Apply>
#>     <Constant dataType="double">4</Constant>
#>     <Constant dataType="double">5</Constant>
#>   </Apply>
#> </Apply>

References