---
title: "Controlled vocabularies"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Controlled vocabularies}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

A *controlled vocabulary* is a predefined set of terms intended to be used to be used in a specific context.
They can solve a number of problems of data standardisation and cleaning in data science.
controller provides functions for working with controlled vocabularies in R. 

This vignette explains the main features of the package.
It introduces the `control()` verb, which recodes values in a vector using a lookup table of preferred and variant terms (a *thesaurus*).

```{r setup, include = FALSE}
library(controller)
```

## The `control()` verb

A common data-tidying problem is standardising variant terms for the same concept.
Imagine we have a dataset that uses a number of different names for shades of the same colour.
As data analysts, we naturally want to recode the data to eliminate this messy creativity, for example using [dplyr::recode()](https://dplyr.tidyverse.org/reference/recode.html):

```{r eg-dplyr}
library(dplyr, warn.conflicts = FALSE)
shades <- c("daffodil", "purple", "magenta", "azure", "navy", "violet")

recode(shades,
       daffodil = "yellow",
       purple = "purple",
       magenta = "pink",
       azure = "blue",
       navy = "blue",
       violet = "purple")
```

But recoding this way can be tedious, especially if there are a large number of terms.
With `control()`, we can instead use a data frame containing a thesaurus to replace the values:

```{r eg-control}
library(controller)
data("colour_thesaurus")

control(shades, colour_thesaurus)
```

By default, `control()` issues a message listing the values that were replaced and a warning for any values that could not be matched in the thesaurus.
These can be suppressed with `quiet = TRUE` and `warn_unmatched = FALSE` respectively.

Use `control_names()` to control the names of an object rather than its values.
This is useful for standardising column names in data frames:

```{r eg-control-names}
df <- data.frame(temp = 20, humid = 65, `wind speed` = 10, date = "2024-01-01")
df

control_names(df, thesaurus = data.frame(
  preferred = c("temperature", "humidity", "wind_speed"),
  variant = c("temp", "humid", "wind speed")
))
```

## Fuzzy matching

`control()` also supports fuzzy matching, removing the need to exhaustively list variants for common causes of differing terminology.
For example, to perform a case insensitive match to the thesaurus:

```{r eg-ci}
shades_ci <- toupper(shades)
control_ci(shades_ci, colour_thesaurus)
```

`control_fuzzy()` goes further by also ignoring differences in word boundaries and character encoding.
Fuzzy boundary matching treats word boundaries like spaces, hyphens, and underscores as equivalent:

```{r eg-boundary}
df <- data.frame(preferred = "foo bar", variant = "foo-bar")
control(c("foo bar", "foo_bar", "foobar"), df,
        fuzzy_boundary = TRUE, quiet = FALSE)
```

Fuzzy encoding matching ignores non-ASCII characters, so plain ASCII input can match variants with diacritics or wrongly encoded characters ('mojibake'):

```{r eg-encoding}
df <- data.frame(preferred = "bar", variant = "fo\u00f6")
control("foo", df, fuzzy_encoding = TRUE, quiet = FALSE)
```

To inspect which type of match was used for each value, use `control_matches()`.
It returns a data frame with a column for each match type, rather than a single vector:

```{r eg-coalesce}
control_matches(shades_ci, colour_thesaurus, case_insensitive = TRUE)
```

## Thesaurus format

A thesaurus is a data frame with two columns containing preferred terms and the variants to be controlled.
Each row maps one variant to one preferred term.
The variants must be unique values – otherwise they cannot be unambiguously matched to a preferred term.

`colour_thesaurus`,^[Compiled by Ingrid Sundberg, <https://web.archive.org/web/20250619164626/https://ingridsundberg.com/2014/02/04/the-color-thesaurus/>.] used in the examples above, is an example of a thesaurus bundled with the package:

```{r thesaurus-preview}
head(colour_thesaurus)
```

Because a thesaurus is simply a data frame, it can be created or read into R with standard tools:

```{r eg-custom-thesaurus}
my_thesaurus <- data.frame(
  preferred = c("feline", "canine"),
  variant = c("cat", "dog")
)

animals <- c("cat", "dog", "parrot")
control(animals, my_thesaurus)
```

Use the `thesaurus_cols` argument to specify which columns of the data frame contain the preferred and variant terms respectively:

```{r eg-thesaurus-cols}
my_wider_thesaurus <- data.frame(
  code = c("FEL", "CAN"),
  label = c("feline", "canine"),
  variant = c("cat", "dog")
)

control(animals, my_wider_thesaurus, thesaurus_cols = c("label", "variant"))
```

The package can also read controlled vocabularies in Historic England's [FISH](https://heritage-standards.org.uk/fish-vocabularies/) (Forum on Information Standards in Heritage) format using `read_fish()`.
The `path` argument can be a local `.zip` file, an uncompressed directory, or a URL:

```{r eg-fish}
nationality_zip <- system.file("extdata", "fish-nationality.zip",
                               package = "controller")
fish_nationality <- read_fish(nationality_zip)
head(fish_nationality)
```

The result is a data frame with columns `preferred` and `term`, ready to use with `control()`.