Getting Started with SWORD

Overview

SWORD (Support vector machine Weighted Oblique Random Decision forests) fits random forests of oblique regression trees. Each internal node uses a Weighted-SVM classifier with a linear kernel to define the oblique splitting hyperplane — a sparse linear combination of predictors — rather than a single axis-aligned threshold.

The base learner is TORS (Tree Oblique for Regression with weighted SVM), a single oblique regression tree introduced in Carta & Frigau (2025). SWORD embeds TORS within a bootstrap aggregation framework, adding predictor subspacing and optional randomisation of the number of features per split to increase tree diversity.

The package provides:


Data

We use the Boston housing dataset from the MASS package (506 observations, 13 numeric predictors, response medv = median house value). The data are split 80/20 into training and test sets.

if (!requireNamespace("MASS", quietly = TRUE))
  stop("Please install the MASS package: install.packages('MASS')")

library(SWORD)
data(Boston, package = "MASS")

set.seed(42)
n     <- nrow(Boston)
train <- sample(n, floor(0.8 * n))
test  <- setdiff(seq_len(n), train)

X_train <- Boston[train, setdiff(names(Boston), "medv")]
y_train <- Boston$medv[train]
X_test  <- Boston[test,  setdiff(names(Boston), "medv")]
y_test  <- Boston$medv[test]

cat("Training set:", nrow(X_train), "obs |",
    "Test set:", nrow(X_test), "obs\n")
#> Training set: 404 obs | Test set: 102 obs

Fitting a single TORS tree

TORS() grows one SVM oblique rotation tree. Key tuning parameters:

Parameter Meaning Default
nmin minimum node size 5
cp complexity threshold for splitting 0.01
n_top_cor top-correlated features fed to the SVM at each node 2
relation feature selection criterion: "Pearson" or "MI" "Pearson"
weight_scheme observation weighting: "scale" or "robust" "scale"
cost_c SVM cost C 1
set.seed(1)
tree <- TORS(X_train, y_train, nmin = 15, cp = 0.02)
print(tree)
#> ---------- TORS (Tree Oblique for Regression with weighted SVM)
#> 
#>            N observations: 404
#>              N predictors: 13
#>                   N nodes: 25 (12 internal, 13 leaves)
#>                 Max depth: 4
#>           Avg obs in leaf: 31.1
#>           Min obs in leaf: 7
#>      Top-correlated feats: 2
#>               Correlation: Pearson
#>             Weight scheme: scale
#> 
#> -----------------------------------------

Tree structure (text)

plot_tors_text() prints the tree as an indented list of nodes. Each internal node shows the oblique hyperplane equation (top-k largest coefficients); each leaf shows the predicted mean.

plot_tors_text(tree, top_k = 3)
#> TORS — text representation
#> ------------------------------------------------------------ 
#> Node 1  n=404  [ - 0.957·rm + 0.282·lstat - 2.837 = 0 ]
#> L-- Node 2  n=193  [ - 1.636·rm + 0.296·lstat - 8.985 = 0 ]
#>   L-- Node 3  n=77  [ - 2.047·rm + 0.226·dis - 13.886 = 0 ]
#>     L-- Node 4  n=30  [ 0.262·ptratio + 0.683·lstat + 7.896 = 0 ]
#>       L-- Leaf  n=18  mean=45.4833
#>       R-- Leaf  n=12  mean=35.8583
#>     R-- Node 7  n=47  [ - 5.578·rm + 1.041·lstat - 33.09 = 0 ]
#>       L-- Leaf  n=26  mean=34.4731
#>       R-- Leaf  n=21  mean=27.4952
#>   R-- Node 10  n=116  [ 0.321·dis - 0.019·rad + 0.978 = 0 ]
#>     L-- Node 11  n=35  [ 1.117·dis + 0.307·lstat + 5.236 = 0 ]
#>       L-- Leaf  n=7  mean=29.8000
#>       R-- Leaf  n=28  mean=23.9643
#>     R-- Leaf  n=81  mean=22.9605
#> R-- Node 15  n=211  [ 0.085·crim + 0.267·lstat + 5.252 = 0 ]
#>   L-- Node 16  n=131  [ - 0.009·black + 0.430·lstat + 3.32 = 0 ]
#>     L-- Node 17  n=82  [ 0.529·dis + 0.254·ptratio + 6.816 = 0 ]
#>       L-- Leaf  n=56  mean=20.9036
#>       R-- Leaf  n=26  mean=19.0615
#>     R-- Leaf  n=49  mean=17.3367
#>   R-- Node 21  n=80  [ 0.013·crim + 0.007·tax + 4.194 = 0 ]
#>     L-- Leaf  n=20  mean=15.5300
#>     R-- Node 23  n=60  [ 5.951·nox + 0.270·lstat + 10.225 = 0 ]
#>       L-- Leaf  n=33  mean=13.8242
#>       R-- Leaf  n=27  mean=10.0074

Tree structure (graph)

plot() on a tors_flat object draws a base-R layout of the tree (node size proportional to the number of observations). If the visNetwork package is installed it renders an interactive zoomable widget instead.

plot(tree)

Effect of cp on tree size

cp is a complexity threshold: a candidate split is accepted only if the relative deviance reduction exceeds cp. Larger values lead to earlier stopping and shallower trees; cp = 0 lets the tree grow until every node reaches nmin.

set.seed(1)
t_deep     <- TORS(X_train, y_train, nmin = 10, cp = 0.000)
t_moderate <- TORS(X_train, y_train, nmin = 10, cp = 0.010)
t_shallow  <- TORS(X_train, y_train, nmin = 10, cp = 0.050)

cp_tbl <- data.frame(
  cp        = c(0.000, 0.010, 0.050),
  n_nodes   = c(t_deep$n_nodes,     t_moderate$n_nodes,     t_shallow$n_nodes),
  n_leaves  = c(sum(t_deep$is_leaf), sum(t_moderate$is_leaf), sum(t_shallow$is_leaf)),
  max_depth = c(max(t_deep$depth),   max(t_moderate$depth),   max(t_shallow$depth))
)
print(cp_tbl, row.names = FALSE)
#>    cp n_nodes n_leaves max_depth
#>  0.00     107       54         8
#>  0.01      37       19         5
#>  0.05      13        7         4

With cp = 0 the tree grows until every node reaches nmin, producing the deepest structure; increasing cp results in more aggressive pruning and shallower trees.

Effect of n_top_cor on split richness

n_top_cor controls how many top-correlated features are passed to the Weighted-SVM at each node. Because TORS splits are always oblique (linear combinations of predictors), the minimum meaningful value is 2; larger values allow the SVM to combine more predictors into denser oblique hyperplanes.

set.seed(1)
t_nc2 <- TORS(X_train, y_train, nmin = 15, cp = 0.02, n_top_cor = 2)
t_nc4 <- TORS(X_train, y_train, nmin = 15, cp = 0.02, n_top_cor = 4)

cat("--- n_top_cor = 2 (default, sparse oblique) ---\n")
#> --- n_top_cor = 2 (default, sparse oblique) ---
plot_tors_text(t_nc2, top_k = 2)
#> TORS — text representation
#> ------------------------------------------------------------ 
#> Node 1  n=404  [ - 0.957·rm + 0.282·lstat - 2.837 = 0 ]
#> L-- Node 2  n=193  [ - 1.636·rm + 0.296·lstat - 8.985 = 0 ]
#>   L-- Node 3  n=77  [ - 2.047·rm + 0.226·dis - 13.886 = 0 ]
#>     L-- Node 4  n=30  [ 0.262·ptratio + 0.683·lstat + 7.896 = 0 ]
#>       L-- Leaf  n=18  mean=45.4833
#>       R-- Leaf  n=12  mean=35.8583
#>     R-- Node 7  n=47  [ - 5.578·rm + 1.041·lstat - 33.09 = 0 ]
#>       L-- Leaf  n=26  mean=34.4731
#>       R-- Leaf  n=21  mean=27.4952
#>   R-- Node 10  n=116  [ 0.321·dis - 0.019·rad + 0.978 = 0 ]
#>     L-- Node 11  n=35  [ 1.117·dis + 0.307·lstat + 5.236 = 0 ]
#>       L-- Leaf  n=7  mean=29.8000
#>       R-- Leaf  n=28  mean=23.9643
#>     R-- Leaf  n=81  mean=22.9605
#> R-- Node 15  n=211  [ 0.085·crim + 0.267·lstat + 5.252 = 0 ]
#>   L-- Node 16  n=131  [ - 0.009·black + 0.430·lstat + 3.32 = 0 ]
#>     L-- Node 17  n=82  [ 0.529·dis + 0.254·ptratio + 6.816 = 0 ]
#>       L-- Leaf  n=56  mean=20.9036
#>       R-- Leaf  n=26  mean=19.0615
#>     R-- Leaf  n=49  mean=17.3367
#>   R-- Node 21  n=80  [ 0.013·crim + 0.007·tax + 4.194 = 0 ]
#>     L-- Leaf  n=20  mean=15.5300
#>     R-- Node 23  n=60  [ 5.951·nox + 0.270·lstat + 10.225 = 0 ]
#>       L-- Leaf  n=33  mean=13.8242
#>       R-- Leaf  n=27  mean=10.0074
cat("\n--- n_top_cor = 4 (richer oblique splits) ---\n")
#> 
#> --- n_top_cor = 4 (richer oblique splits) ---
plot_tors_text(t_nc4, top_k = 4)
#> TORS — text representation
#> ------------------------------------------------------------ 
#> Node 1  n=404  [ - 0.031·indus - 0.934·rm + 0.172·ptratio + 0.294·lstat + 0.241 = 0 ]
#> L-- Node 2  n=181  [ - 1.278·rm + 0.203·dis + 0.162·ptratio + 0.342·lstat - 2.681 = 0 ]
#>   L-- Node 3  n=70  [ - 0.261·crim - 1.894·rm + 0.207·dis + 0.405·lstat - 11.347 = 0 ]
#>     L-- Node 4  n=22  [ 0.837·rm - 0.112·rad + 0.511·ptratio - 0.012·black + 9.77 = 0 ]
#>       L-- Leaf  n=12  mean=48.2417
#>       R-- Leaf  n=10  mean=40.9100
#>     R-- Node 7  n=48  [ - 0.086·indus - 5.187·rm - 0.000·tax - 0.006·black - 39.064 = 0 ]
#>       L-- Leaf  n=27  mean=34.0519
#>       R-- Leaf  n=21  mean=30.1143
#>   R-- Node 10  n=111  [ 0.151·crim - 0.328·rad + 0.011·tax + 0.107·lstat + 2.842 = 0 ]
#>     L-- Node 11  n=66  [ 0.038·crim - 0.032·age + 0.030·rad + 0.041·black + 14.063 = 0 ]
#>       L-- Leaf  n=13  mean=29.3231
#>       R-- Leaf  n=53  mean=24.3283
#>     R-- Leaf  n=45  mean=22.6933
#> R-- Node 15  n=223  [ 0.066·crim + 0.046·age + 0.001·tax + 0.277·lstat + 10.061 = 0 ]
#>   L-- Node 16  n=132  [ - 2.085·nox + 0.042·age - 0.007·black + 0.138·lstat + 1.603 = 0 ]
#>     L-- Leaf  n=80  mean=20.4100
#>     R-- Leaf  n=52  mean=18.2769
#>   R-- Node 19  n=91  [ 0.039·crim - 0.340·dis + 0.009·tax + 0.237·lstat + 10.901 = 0 ]
#>     L-- Leaf  n=54  mean=15.0241
#>     R-- Leaf  n=37  mean=10.5054

With n_top_cor = 2 each split combines two predictors; with n_top_cor = 4 the Weighted-SVM can involve up to four predictors per split, producing denser oblique hyperplanes at the cost of higher per-node fitting time.


Fitting a SWORD forest

SWORD() fits a random forest of TORS trees with bootstrap sampling. Setting oob = TRUE (the default) accumulates out-of-bag predictions at each iteration, which lets you monitor convergence without a separate validation set.

set.seed(42)
forest <- SWORD(X_train, y_train,
                m       = 50,
                nmin    = 10,
                cp      = 0.01,
                oob     = TRUE,
                verbose = FALSE)
print(forest)
#> ---------- SWORD (Support vector machine Weighted Oblique Random Decision forest)
#> 
#>            N observations: 404
#>              N predictors: 13
#>                   N trees: 50
#>      Predictors per split: 13
#>      Top-correlated feats: random [2, 13]
#>       Avg leaves per tree: 12.920
#>           Min obs in leaf: 2
#>             OOB stat type: RMSE / R²
#>            OOB stat value: 4.2225 / 0.7764
#>             Weight scheme: scale
#>               Correlation: Pearson
#> 
#> -----------------------------------------

OOB diagnostics

Convergence curve

plot(forest, which = "oob") plots the OOB MSE as a function of the number of trees. The dashed line marks the minimum; check that it has levelled off before using the forest for inference.

plot(forest, which = "oob")

OOB predicted vs. observed

plot_oob_fit() plots the out-of-bag predictions against the true training responses. Points on the identity line indicate perfect fit; the annotation reports RMSE, MAE, and R.

plot_oob_fit(forest, y_train)


Prediction on the test set

y_hat <- predict(forest, X_test)

rmse <- sqrt(mean((y_hat - y_test)^2))
mae  <- mean(abs(y_hat - y_test))
r2   <- cor(y_hat, y_test)^2

cat(sprintf("Test RMSE : %.3f\n", rmse))
#> Test RMSE : 3.511
cat(sprintf("Test MAE  : %.3f\n", mae))
#> Test MAE  : 2.240
cat(sprintf("Test R^2  : %.3f\n", r2))
#> Test R^2  : 0.891

The predict() method also accepts a data.frame that contains extra columns (e.g. the response), so you can pass the original data frame directly:

y_hat2 <- predict(forest, Boston[test, ])  # medv column is silently dropped
all.equal(y_hat, y_hat2)
#> [1] TRUE

Variable importance

VI_SWORD() computes the OIW-VI (Oblique Impurity-Weighted Variable Importance): at each internal node, the deviance reduction is distributed among the predictors involved in the oblique split proportionally to their normalised absolute Weighted-SVM coefficients; these scores are then averaged across all trees. plot_vi_sword() draws a horizontal barplot sorted by importance.

vi <- VI_SWORD(forest)
print(round(vi, 2))
#>   lstat      rm ptratio     rad     tax     dis     nox    crim     age   indus 
#> 9283.94 4907.46 2487.42 1841.03 1742.81 1721.06 1529.35 1503.95 1058.03 1014.01 
#>   black      zn    chas 
#>  652.43  468.43  455.07
plot_vi_sword(forest, top_n = 13)


Partial dependence plots

pdp_sword() computes the marginal effect of a single predictor by fixing it to a grid of values and averaging forest predictions over the full training set (Friedman 2001). A rug of observed values is added by default.

old_par <- par(mfrow = c(1, 2), mar = c(4, 4, 3, 1))
pdp_sword(forest, X_train, "lstat", main = "PDP: lstat")
pdp_sword(forest, X_train, "rm",    main = "PDP: rm")

par(old_par)

Top-4 predictors in a 2x2 grid:

top4 <- names(vi)[1:4]
old_par <- par(mfrow = c(2, 2), mar = c(4, 4, 3, 1))
for (v in top4)
  pdp_sword(forest, X_train, v, main = paste("PDP:", v))

par(old_par)

Parameter sensitivity in SWORD

Four parameters most likely to affect predictive accuracy are relation, weight_scheme, rf_var, and rand_ntopcor. In this section we fit compact forests (m = 20, nmin = 10, cp = 0.01) varying each parameter in turn and compare test-set RMSE on the same Boston hold-out.

Feature selection criterion: relation

relation controls how the top-correlated features are identified at each node: "Pearson" uses the absolute Pearson correlation with the node residuals; "MI" uses mutual information, which can capture non-linear associations between each predictor and the response.

set.seed(42)
f_pearson <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                   relation = "Pearson", verbose = FALSE)
set.seed(42)
f_mi      <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                   relation = "MI",      verbose = FALSE)

cat(sprintf("Pearson  — test RMSE: %.3f\n", rmse_f(y_test, predict(f_pearson, X_test))))
#> Pearson  — test RMSE: 3.586
cat(sprintf("MI       — test RMSE: %.3f\n", rmse_f(y_test, predict(f_mi,      X_test))))
#> MI       — test RMSE: 3.592

Both criteria are competitive on continuous predictors; "MI" may offer an advantage when predictor–response associations are strongly non-linear.

Observation weighting: weight_scheme

weight_scheme determines how observations are weighted when fitting the Weighted-SVM at each node. Both schemes assign higher weight to observations whose response is far from the local centre, guiding the SVM to concentrate on the most informative thresholds. The two formulas differ in how the centre and spread are estimated:

\[ w_i^{\text{scale}} = \frac{|y_i - \bar{y}|}{s_y}, \qquad w_i^{\text{robust}} = \frac{|y_i - \text{median}(y)|}{\text{MAD}(y)} \]

where \(s_y\) is the standard deviation and \(\text{MAD}(y) = \text{median}(|y_i - \text{median}(y)|)\). Because "robust" centres on the median and scales by the MAD rather than the mean and standard deviation, it is less influenced by extreme values in the response.

set.seed(42)
f_scale  <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                  weight_scheme = "scale",  verbose = FALSE)
set.seed(42)
f_robust <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                  weight_scheme = "robust", verbose = FALSE)

cat(sprintf("scale    — test RMSE: %.3f\n", rmse_f(y_test, predict(f_scale,  X_test))))
#> scale    — test RMSE: 3.586
cat(sprintf("robust   — test RMSE: %.3f\n", rmse_f(y_test, predict(f_robust, X_test))))
#> robust   — test RMSE: 3.556

"robust" is recommended when the training response contains outliers or extreme values that would inflate \(s_y\) and distort the "scale" weights.

Feature subsampling per split: rf_var

rf_var sets how many predictors are randomly sub-sampled at each node before the top-n_top_cor correlated ones are selected for the SVM. Reducing rf_var increases tree decorrelation — analogously to mtry in standard random forests. Three common choices with p = 13 predictors:

set.seed(42)
f_full <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                rf_var = p,               verbose = FALSE)  # all predictors
set.seed(42)
f_half <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                rf_var = floor(p / 2),    verbose = FALSE)  # half
set.seed(42)
f_sqrt <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                rf_var = floor(sqrt(p)),  verbose = FALSE)  # sqrt rule
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero

cat(sprintf("rf_var = %2d (all)        — test RMSE: %.3f\n",
            p,               rmse_f(y_test, predict(f_full, X_test))))
#> rf_var = 13 (all)        — test RMSE: 3.586
cat(sprintf("rf_var = %2d (floor p/2)  — test RMSE: %.3f\n",
            floor(p / 2),   rmse_f(y_test, predict(f_half, X_test))))
#> rf_var =  6 (floor p/2)  — test RMSE: 3.592
cat(sprintf("rf_var = %2d (floor sqrt) — test RMSE: %.3f\n",
            floor(sqrt(p)), rmse_f(y_test, predict(f_sqrt, X_test))))
#> rf_var =  3 (floor sqrt) — test RMSE: 4.488

Unlike axis-aligned forests, where reducing mtry decorrelates trees and often lowers test error, in SWORD a larger candidate pool consistently improves accuracy: allowing the Weighted-SVM to screen more predictors at each node leads to better oblique splits without sacrificing tree diversity (which is already provided by bootstrap sampling and the random- mechanism). Restricting rf_var limits the quality of the oblique hyperplane and in practice tends to worsen RMSE. The default (rf_var = p, all predictors) is therefore recommended.

Randomised top-feature cardinality: rand_ntopcor

When rand_ntopcor = TRUE (the SWORD default) the number of top-correlated features passed to the SVM at each node is itself drawn at random from , introducing an extra source of tree diversity beyond bootstrap sampling. Setting it to FALSE fixes that count at n_top_cor for every split.

set.seed(42)
f_rnd_t <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                 rand_ntopcor = TRUE,  verbose = FALSE)
set.seed(42)
f_rnd_f <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
                 rand_ntopcor = FALSE, verbose = FALSE)

cat(sprintf("rand_ntopcor = TRUE  — test RMSE: %.3f\n",
            rmse_f(y_test, predict(f_rnd_t, X_test))))
#> rand_ntopcor = TRUE  — test RMSE: 3.586
cat(sprintf("rand_ntopcor = FALSE — test RMSE: %.3f\n",
            rmse_f(y_test, predict(f_rnd_f, X_test))))
#> rand_ntopcor = FALSE — test RMSE: 3.766

Randomisation usually helps when m is large enough; for very small forests it can occasionally hurt by introducing excess variance.

Summary comparison

#>      Parameter                  Value Test_RMSE
#>       relation      Pearson (default)     3.586
#>       relation                     MI     3.592
#>  weight_scheme        scale (default)     3.586
#>  weight_scheme                 robust     3.556
#>         rf_var p = 13 — all (default)     3.586
#>         rf_var         floor(p/2) = 6     3.592
#>         rf_var     floor(sqrt(p)) = 3     4.488
#>   rand_ntopcor         TRUE (default)     3.586
#>   rand_ntopcor                  FALSE     3.766

Categorical and character predictors

SWORD handles factor and character columns automatically: they are one-hot-encoded internally, new factor levels in predict() are mapped to all-zero dummies, and VI_SWORD() sums dummy-column importances back to the original variable name.

data(diamonds, package = "ggplot2")
set.seed(7)
idx   <- sample(nrow(diamonds), 2000)
X_dia <- diamonds[idx, c("carat","cut","color","clarity",
                          "depth","table","x","y","z")]
y_dia <- diamonds$price[idx]

forest_dia <- SWORD(X_dia, y_dia, m = 50, oob = TRUE, verbose = FALSE)
print(forest_dia)

# VI grouped by original variable (not by dummy column)
VI_SWORD(forest_dia)

# PDP for an ordered factor
pdp_sword(forest_dia, X_dia, "cut")

Formula interface

Both TORS() and SWORD() accept an R formula with a data argument:

df <- cbind(X_train, medv = y_train)

tree2   <- TORS(medv ~ ., data = df, nmin = 15, cp = 0.02)
forest2 <- SWORD(medv ~ ., data = df, m = 50, verbose = FALSE)
predict(forest2, Boston[test, ])

Summary

Task Function
Fit one tree TORS(X, y)
Fit a forest SWORD(X, y, m = 100)
Predict predict(forest, newdata)
OOB convergence plot(forest, which = "oob")
OOB scatter plot_oob_fit(forest, y)
OIW-VI (variable importance) VI_SWORD(forest) / plot_vi_sword(forest)
Partial dependence pdp_sword(forest, X, "varname")
Tree text print plot_tors_text(tree)
Tree graph plot(tree)

References

Carta, A. and Frigau, L. (in press). Oblique random forests for regression via weighted support vector machine. Advances in Data Analysis and Classification.

Carta, A. and Frigau, L. (2025). Tree oblique for regression with weighted support vector machine. Computational Statistics, 40, 5257–5291. https://doi.org/10.1007/s00180-025-01647-w

Yang, X., Song, Q. and Cao, A. (2005). Weighted support vector machine for data classification. In Proceedings of the 2005 IEEE International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/IJCNN.2005.1555965

Xu, T. et al. (2024). WeightSVM: Subject/Instance Weighted Support Vector Machines. R package version 1.7-16. https://doi.org/10.32614/CRAN.package.WeightSVM