Package {tidylda}

Type:	Package
Title:	Latent Dirichlet Allocation Using 'tidyverse' Conventions
Version:	0.1.0
Description:	Implements an algorithm for Latent Dirichlet Allocation (LDA), Blei et al. (2003) https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf, using style conventions from the 'tidyverse', Wickham et al. (2019)<doi:10.21105/joss.01686>, and 'tidymodels', Kuhn et al.https://tidymodels.github.io/model-implementation-principles/. Fitting is done via 'warpLDA', a Metropolis-Hastings sampler, Chen et al. (2016) <doi:10.48550/arXiv.1510.08628>. Also implements several novel features for LDA such as guided models and transfer learning.
License:	MIT + file LICENSE
URL:	https://github.com/TommyJones/tidylda/
BugReports:	https://github.com/TommyJones/tidylda/issues
Depends:	R (≥ 3.5.0)
Imports:	dplyr, generics, gtools, Matrix, methods, mvrsquared (≥ 0.1.0), Rcpp (≥ 1.0.2), rlang, stats, stringr, tibble, tidyr, tidytext
Suggests:	ggplot2, knitr, parallel, quanteda, testthat, tm, slam, spelling, covr, rmarkdown
LinkingTo:	Rcpp, RcppArmadillo, RcppProgress, RcppThread
Encoding:	UTF-8
Language:	en-US
LazyData:	true
VignetteBuilder:	knitr
Config/roxygen2/version:	8.0.0
NeedsCompilation:	yes
Packaged:	2026-08-27 20:20:23 UTC; twj22
Author:	Tommy Jones [aut, cre], Brendan Knapp [ctb], Barum Park [ctb]
Maintainer:	Tommy Jones <jones.thos.w@gmail.com>
Repository:	CRAN
Date/Publication:	2026-08-28 07:00:09 UTC

Latent Dirichlet Allocation Using 'tidyverse' Conventions

Description

Implements an algorithm for Latent Dirichlet Allocation (LDA) using style conventions from the 'tidyverse' and specifically 'tidymodels'. Also implements several novel features for LDA such as guided models and transfer learning.

Fitting uses warpLDA (Chen et al., 2016, doi:10.48550/arXiv.1510.08628), a Metropolis-Hastings sampler that alternates document-ordered and word-ordered passes over the corpus so that each pass touches only a small, cache-resident working set. It replaced the collapsed Gibbs sampler in version 0.1.0. Sampling is multithreaded via the threads argument and results do not depend on the thread count.

Options

tidylda.max_result_size caps the size of the object posterior.tidylda and tidy.tidylda will build, in bytes. Both return one row per cell of a matrix that grows with topics times vocabulary, so a plausible-looking call can ask for billions of rows; above the cap they raise an error naming the size and a smaller alternative rather than exhausting the session. Defaults to 1024^3 (1 GB). Raise it with options(tidylda.max_result_size = 4 * 1024^3).

Author(s)

Maintainer: Tommy Jones jones.thos.w@gmail.com (ORCID)

Authors:

• Tommy Jones jones.thos.w@gmail.com (ORCID)

Other contributors:

• Brendan Knapp brendan.g.knapp@gmail.com (ORCID) [contributor]

• Barum Park baruuum@gmail.com [contributor]

See Also

Useful links:

• https://github.com/TommyJones/tidylda/

• Report bugs at https://github.com/TommyJones/tidylda/issues

Augment method for tidylda objects

Description

augment appends observation level model outputs.

Usage

## S3 method for class 'tidylda'
augment(
  x,
  data,
  type = c("class", "prob"),
  document_col = "document",
  term_col = "term",
  ...
)

Arguments

Details

The key statistic for augment is P(topic | document, token) = P(topic | token) * P(token | document). P(topic | token) are the entries of the 'lambda' matrix in the tidylda object passed with x. P(token | document) is taken to be the frequency of each token normalized within each document.

Value

augment returns a tidy tibble containing one row per document-token pair, with one or more columns appended, depending on the value of type.

If type = 'prob', then one column per topic is appended. Its value is P(topic | document, token).

If type = 'class', then the most-probable topic for each document-token pair is returned. If multiple topics are equally probable, then the topic with the smallest index is returned by default.

Calculate a matrix whose rows represent P(topic_i|tokens)

Description

Use Bayes' rule to get P(topic|token) from the estimated parameters of a probabilistic topic model.This resulting "lambda" matrix can be used for classifying new documents in a frequentist context and supports augment.

Usage

calc_lambda(beta, theta, p_docs = NULL, correct = TRUE)

Arguments

Value

Returns a matrix whose rows correspond to topics and whose columns correspond to tokens. The i,j entry corresponds to P(topic_i|token_j)

Calculate R-squared for a tidylda Model

Description

Formats inputs and hands off to calc_rsquared

Usage

calc_lda_r2(dtm, theta, beta, threads)

Arguments

Value

Numeric scalar between negative infinity and 1

Probabilistic coherence of topics

Description

Calculates the probabilistic coherence of a topic or topics. This approximates semantic coherence or human understandability of a topic.

Usage

calc_prob_coherence(beta, data, m = 5)

Arguments

Details

For each pair of words {a, b} in the top M words in a topic, probabilistic coherence calculates P(b|a) - P(b), where {a} is more probable than {b} in the topic. For example, suppose the top 4 words in a topic are {a, b, c, d}. Then, we calculate 1. P(a|b) - P(b), P(a|c) - P(c), P(a|d) - P(d) 2. P(b|c) - P(c), P(b|d) - P(d) 3. P(c|d) - P(d) All 6 differences are averaged together.

Value

Returns an object of class numeric corresponding to the probabilistic coherence of the input topic(s).

Examples

# Load a pre-formatted dtm and topic model
data(nih_sample_dtm)

# fit a model
set.seed(12345)
model <- tidylda(
  data = nih_sample_dtm[1:20, ], k = 5,
  iterations = 100, burnin = 50
)

calc_prob_coherence(beta = model$beta, data = nih_sample_dtm, m = 5)

Refuse to build a result that cannot fit in memory

Description

posterior.tidylda and tidy.tidylda both return one row per cell of something that grows as k * V, so an innocuous-looking call can ask for an object of billions of rows. Previously such a call simply exhausted the session. This raises an error that says how large the result would be and what to do instead.

The ceiling is a fixed 1 GB, which options(tidylda.max_result_size = <bytes>) can raise. Fixed rather than derived from free memory, so that behavior is reproducible across machines; it is meant to catch a pathological request, not to track RAM.

Usage

check_result_size(n_rows, n_cols, what, suggestion)

Arguments

Value

invisible(NULL), or an error.

Convert various things to a dgCMatrix to work with various functions and methods

Description

Presently, tidylda makes heavy usage of the dgCMatrix class. However, a user may have created a DTM (or TCM) in one of several classes. Since data could be in several formats, this function converts them to a dgCMatrix before passing them along.

Usage

convert_dtm(dtm)

Arguments

Value

an object of class dgCMatrix

Read counts$Cv in the orientation this version expects

Description

D17 changed the exported word-topic counts from topics-by-words to words-by-topics, matching the orientation the engine holds them in. Models fitted by earlier versions carry the old shape, and both refit.tidylda and posterior.tidylda index this matrix directly.

A wrong orientation on the transfer-learning path would corrupt \omega_k^{*(t)} rather than fail loudly, so rather than trust a length mismatch to error, detect the shape against beta – which is topics-by-words in every version – and transpose an old object on read.

Usage

counts_cv(object)

Arguments

Value

object$counts$Cv as a words-by-topics matrix.

Materialize eta as a topics-by-tokens matrix

Description

Since D20, format_eta leaves a scalar prior as a scalar rather than expanding it to k by Nv. Call this where a full matrix is genuinely required. Most call sites do not need one – arithmetic against a scalar recycles correctly, and the sampler takes the scalar directly.

Usage

eta_matrix(eta, k, Nv)

Arguments

Value

a numeric matrix with k rows and Nv columns.

Row sums of eta without materializing it

Description

rowSums(eta_matrix(...)) allocates a dense k by Nv matrix only to collapse it immediately. That is 8 GB at k = 1000, Nv = 1e6. A scalar prior gives every row the same sum, so it needs no matrix at all; a matrix prior is summed directly, since eta_matrix() would have returned it unchanged anyway.

NOT BIT-IDENTICAL to the materialized form, and deliberately so. rowSums() accumulates in long double, so it returns 1500.0000000000002274 where Nv * eta returns exactly 1500 — a relative difference of 1.5e-16. The engine stores eta as float (D5), whose resolution is nine orders of magnitude coarser, so the difference is erased before the sampler sees it. The only trace is the 16th significant digit of a refitted model's eta slot.

Usage

eta_row_sums(eta, k, Nv)

Arguments

Value

a numeric vector of length k.

Fit an LDA model with the warpLDA sampler

Description

Metropolis-Hastings replacement for fit_lda_c. Phase 2: single-threaded, scalar eta only.

Usage

fit_lda_warp(
  dtm_in,
  Cd_start,
  alpha_in,
  eta_in,
  iterations,
  burnin,
  calc_likelihood,
  Beta_in,
  freeze_topics = FALSE,
  likelihood_every = 10L,
  mh_steps = 1L,
  threads = 1L,
  verbose = TRUE
)

Arguments

Value

Returns a list of counts and diagnostics. Cd and Cd_mean are documents by topics; Cv and Cv_mean are words by topics (D17). Only the pair the caller can use is materialized: Cd/Cv when burnin is -1, Cd_mean/ Cv_mean otherwise. The other pair comes back 0 x 0.

Format alpha for input into the sampler

Description

There are a bunch of ways users could format alpha but the C++ sampler in fit_lda_warp only takes it one way. This function does the appropriate formatting. It also returns errors if the user input a malformatted alpha.

Usage

format_alpha(alpha, k)

Arguments

Value

Returns a list with two elements: alpha and alpha_class. alpha is the post-formatted version of alpha in the form of a k-length numeric vector. alpha_class is a character denoting whether or not the user-supplied alpha was a "scalar" or "vector".

Format eta for input into the sampler

Description

There are a bunch of ways users could format eta but the C++ sampler in fit_lda_warp only takes it one way. This function does the appropriate formatting. It also returns errors if the user input a malformatted eta.

Usage

format_eta(eta, k, Nv)

Arguments

Value

Returns a list with two elements: eta and eta_class. eta is the post-formatted version of eta in the form of a k by Nv numeric matrix. eta_class is a character denoting whether or not the user-supplied eta was a "scalar", "vector", or "matrix".

Generate a sample of LDA posteriors

Description

Helper function called by both posterior.tidylda and predict.tidylda to generate samples from the posterior.

Usage

generate_sample(dir_par, matrix, times)

Arguments

Value

Returns a tibble with one row per parameter per sample.

Glance method for tidylda objects

Description

glance constructs a single-row summary "glance" of a tidylda topic model.

Usage

## S3 method for class 'tidylda'
glance(x, ...)

Arguments

Value

glance returns a one-row tibble with the following columns:

num_topics: the number of topics in the model num_documents: the number of documents used for fitting num_tokens: the number of tokens covered by the model iterations: number of total sampling iterations run burnin: number of burn-in iterations run

Examples

dtm <- nih_sample_dtm

lda <- tidylda(data = dtm, k = 10, iterations = 100, burnin = 75)

glance(lda)

Prepare the priors the sampler initializes from

Description

Implementing seeded (or guided) LDA models and transfer learning means that we can't initialize topics with a uniform-random start. This function prepares the two matrices the sampler needs in order to draw an informed starting assignment: beta_initial, giving P(token|topic), and Cd_start, the expected number of tokens each topic accounts for in each document. In the event that you aren't using fancy seeding or transfer learning, this makes a random initialization by sampling from Dirichlet distributions parameterized by priors alpha and eta.

The per-token work of building the token structure and sampling each token's starting topic happens inside fit_lda_warp, so nothing proportional to the token count crosses the R/C++ boundary.

Usage

initialize_topic_counts(
  dtm,
  k,
  alpha,
  eta,
  beta_initial = NULL,
  theta_initial = NULL,
  freeze_topics = FALSE,
  threads = 1,
  ...
)

Arguments

Value

Returns a list with two elements, both of which the engine consumes directly:

beta_initial is a numeric matrix with one row per topic and one column per token, giving P(token|topic) to initialize from. Supplied by the caller for updates and predictions; sampled from eta otherwise.

Cd_start is a numeric matrix, documents by topics, holding theta_initial * rowSums(dtm) — the expected number of tokens each topic accounts for in each document.

Together these define the informed initialization: the engine samples each token's starting topic from P(z) proportional to beta_initial[k, v] * (Cd_start[d, k] + alpha[k]), in log space. That per-token work happens in C++ (see fit_lda_warp), so nothing proportional to the token count crosses the R/C++ boundary.

Construct a new object of class tidylda

Description

Since all three of tidylda, refit.tidylda, and predict.tidylda call fit_lda_warp, we need a way to format the resulting posteriors and other user-facing objects consistently. This function does that.

Usage

new_tidylda(
  lda,
  dtm,
  burnin,
  is_prediction = FALSE,
  alpha = NULL,
  eta = NULL,
  optimize_alpha = NULL,
  calc_r2 = NULL,
  calc_likelihood = NULL,
  call = NULL,
  threads
)

Arguments

Value

Returns an S3 object of class tidylda with the following slots:

beta is a numeric matrix whose rows are the posterior estimates of P(token|topic)

theta is a numeric matrix whose rows are the posterior estimates of P(topic|document)

lambda is a numeric matrix whose rows are the posterior estimates of P(topic|token), calculated using Bayes's rule. See calc_lambda.

alpha is the prior for topics over documents. It is what the user passed when calling tidylda, formatted as a k-length numeric vector; the sampler does not modify it.

eta is the prior for tokens over topics. This is what the user passed when calling tidylda: a numeric scalar stays a scalar, and a matrix prior is a k by ncol(dtm) matrix.

counts is a list of two matrices holding the token-topic counts the sampler ended on. Cd is a dense matrix of documents by topics. Cv is a sparse dgCMatrix-class of tokens by topics, labeled with the model's vocabulary and topic names. Both are topics-in-columns, so Cd aligns with theta and Cv with t(beta). If burn-in iterations were used these are averages over the post-burn-in iterations, and are therefore not integers.

Cd is deliberately dense: it is 38-81 percent nonzero, where a sparse form saves nothing and can cost 20 percent. Cv is 8-23 percent nonzero and roughly 3 times smaller sparse.

NOTE: as of version 0.1.0 Cv is tokens by topics and sparse; it was topics by tokens and dense previously.

summary is the result of a call to summarize_topics

call is the result of match.call called at the top of tidylda

log_likelihood is a tibble with three columns, evaluated every likelihood_every-th iteration. iteration is the iteration number. log_likelihood is P(tokens | \theta, \beta), the plug-in likelihood of the data under the current parameter estimates. log_joint is P(tokens, topics | \alpha, \eta), the collapsed joint, with theta and beta integrated out. See tidylda for which to use when. This slot is only populated if calc_likelihood = TRUE

r2 is a numeric scalar resulting from a call to calc_rsquared. This slot only populated if calc_r2 = TRUE

Note

In general, the arguments of this function should be what the user passed when calling tidylda.

burnin is used only to determine whether or not burn in iterations were used when fitting the model. If burnin > -1 then posteriors are calculated using lda$Cd_mean and lda$Cv_mean respectively. Otherwise, posteriors are calculated using lda$Cd_mean and lda$Cv_mean.

The class of call isn't checked. It's just passed through to the object returned by this function. Might be useful if you are using this function for troubleshooting or something.

Abstracts and metadata from NIH research grants awarded in 2014

Description

This dataset holds information on research grants awarded by the National Institutes of Health (NIH) in 2014. The data set was downloaded in approximately January of 2015. It includes both 'projects' and 'abstracts' files.

Usage

data("nih_sample")

Format

For nih_sample, a tibble of 100 randomly-sampled grants' abstracts and metadata. For nih_sample_dtm, a dgCMatrix-class representing the document term matrix of abstracts from 100 randomly-sampled grants.

Source

National Institutes of Health ExPORTER https://reporter.nih.gov/exporter

Pad a document term matrix with empty columns for missing vocabulary

Description

Both refit.tidylda and predict.tidylda align a new DTM against a model's vocabulary by appending all-zero columns for terms the data lacks. This is that operation, in one place.

THE FILLER MUST BE SPARSE. Until 0.1.0 refit() built it with matrix(0, ...), a dense allocation of nrow(dtm) by length(add) doubles — 5.4 GB on a 48,508-document corpus with 15,000 model-only terms, enough to exhaust a 32 GB session before sampling began. The waste was total, since cbind() of a sparse and a dense matrix returns a dgCMatrix regardless: the dense block existed only as an argument. predict() always did this correctly, which is why the two are now one function.

Usage

pad_vocabulary(dtm, add)

Arguments

Value

dtm with one all-zero column per entry of add, appended in order, with row and column names preserved.

Draw from the marginal posteriors of a tidylda topic model

Description

Sample from the marginal posteriors of a tidylda topic model. This is useful for quantifying uncertainty around the parameters of beta or theta.

Usage

posterior(x, ...)

## S3 method for class 'tidylda'
posterior(x, matrix, which, times, ...)

Arguments

Value

posterior returns a tibble with one row per parameter per sample.

Returns a data frame where each row is a single sample from the posterior. Each column is the distribution over a single parameter. The variable var is a facet for subsetting by document (for theta) or topic (for beta).

References

Heinrich, G. (2005) Parameter estimation for text analysis. Technical report. Archived copy (arbylon.net no longer resolves; this is the Internet Archive's copy.)

Examples

# load some data
data(nih_sample_dtm)

# fit a model
set.seed(12345)

m <- tidylda(
  data = nih_sample_dtm[1:20, ], k = 5,
  iterations = 200, burnin = 175
)

# sample from the marginal posterior corresponding to topic 1
t1 <- posterior(
  x = m,
  matrix = "beta",
  which = 1,
  times = 100  
)

# sample from the marginal posterior corresponding to documents 5 and 6
d5 <- posterior(
  x = m,
  matrix = "theta",
  which = c(5, 6),
  times = 100
)

Get predictions from a Latent Dirichlet Allocation model

Description

Obtains predictions of topics for new documents from a fitted LDA model

Usage

## S3 method for class 'tidylda'
predict(
  object,
  new_data,
  type = c("prob", "class", "distribution"),
  method = c("mh", "dot", "gibbs"),
  iterations = NULL,
  burnin = -1,
  no_common_tokens = c("default", "zero", "uniform"),
  times = 100,
  threads = 1,
  verbose = TRUE,
  mh_steps = 1,
  ...
)

Arguments

Details

If predict.tidylda encounters documents that have no tokens in common with the model in object it will engage in one of three behaviors based on the setting of no_common_tokens.

default (the default) sets all topics to 0 for offending documents. This enables continued computations downstream in a way that NA would not. However, if no_common_tokens == "default", then predict.tidylda will emit a warning for every such document it encounters.

zero has the same behavior as default but it emits a message instead of a warning.

uniform sets all topics to 1/k for every topic for offending documents. it does not emit a warning or message.

Value

type gives different outputs depending on whether the user selects "prob", "class", or "distribution". If "prob", the default, returns a a "theta" matrix with one row per document and one column per topic. If "class", returns a vector with the topic index of the most likely topic in each document. If "distribution", returns a tibble with one row per parameter per sample. Number of samples is set by the times argument.

Examples

# load some data
data(nih_sample_dtm)

# fit a model
set.seed(12345)

m <- tidylda(
  data = nih_sample_dtm[1:20, ], k = 5,
  iterations = 200, burnin = 175
)

str(m)

# predict on held-out documents using Metropolis-Hastings "fold in"
p1 <- predict(m, nih_sample_dtm[21:100, ],
  method = "mh",
  iterations = 200, burnin = 175
)

# predict on held-out documents using the dot product
p2 <- predict(m, nih_sample_dtm[21:100, ], method = "dot")

# compare the methods
barplot(rbind(p1[1, ], p2[1, ]), beside = TRUE, col = c("red", "blue"))

# predict classes on held out documents
p3 <- predict(m, nih_sample_dtm[21:100, ],
  method = "mh",
  type = "class",
  iterations = 100, burnin = 75
)

# predict distribution on held out documents
p4 <- predict(m, nih_sample_dtm[21:100, ],
  method = "mh",
  type = "distribution",
  iterations = 100, burnin = 75,
  times = 10
)

Print Method for tidylda

Description

Print a summary for objects of class tidylda

Usage

## S3 method for class 'tidylda'
print(x, digits = max(3L, getOption("digits") - 3L), n = 5, ...)

Arguments

Value

Silently returns x

Examples

dtm <- nih_sample_dtm

lda <- tidylda(data = dtm, k = 10, iterations = 100)

print(lda)

lda

print(lda, digits = 2)

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

generics

augment(), glance(), refit(), tidy()

Update a Latent Dirichlet Allocation topic model

Description

Update an LDA model using warpLDA's Metropolis-Hastings sampler.

Usage

## S3 method for class 'tidylda'
refit(
  object,
  new_data,
  iterations = NULL,
  burnin = -1,
  prior_weight = 1,
  additional_k = 0,
  additional_eta_sum = 250,
  optimize_alpha = FALSE,
  calc_likelihood = FALSE,
  calc_r2 = FALSE,
  return_data = FALSE,
  threads = 1,
  verbose = TRUE,
  likelihood_every = 10,
  mh_steps = 1,
  ...
)

Arguments

Details

refit allows you to (a) update the probabilities (i.e. weights) of a previously-fit model with new data or additional iterations and (b) optionally use beta of a previously-fit LDA topic model as the eta prior for the new model. This is tuned by setting beta_as_prior = FALSE or beta_as_prior = TRUE respectively.

prior_weight tunes how strong the base model is represented in the prior. If prior_weight = 1, then the tokens from the base model's training data have the same relative weight as tokens in new_data. In other words, it is like just adding training data. If prior_weight is less than 1, then tokens in new_data are given more weight. If prior_weight is greater than 1, then the tokens from the base model's training data are given more weight.

If prior_weight is NA, then the new eta is equal to eta from the old model, with new tokens folded in. (For handling of new tokens, see below.) Effectively, this just controls how the sampler initializes (described below), but does not give prior weight to the base model.

Instead of initializing token-topic assignments in the manner for new models (see tidylda), the update initializes in 2 steps:

First, topic-document probabilities (i.e. theta) are obtained by a call to predict.tidylda using method = "dot" for the documents in new_data. Next, both beta and theta are passed to an internal function, initialize_topic_counts, which assigns topics to tokens in a manner approximately proportional to the posteriors and executes a single sampling iteration.

refit handles the addition of new vocabulary by adding a flat prior over new tokens. Specifically, each entry in the new prior is equal to the 10th percentile of eta from the old model. The resulting model will have the total vocabulary of the old model plus any new vocabulary tokens. In other words, after running refit.tidylda ncol(beta) >= ncol(new_data) where beta is from the new model and new_data is the additional data.

You can add additional topics by setting the additional_k parameter to an integer greater than zero. New entries to alpha have a flat prior equal to the median value of alpha in the old model. (Note that if alpha itself is a flat prior, i.e. scalar, then the new topics have the same value for their prior.) New entries to eta have a shape from the average of all previous topics in eta and scaled by additional_eta_sum.

Value

Returns an S3 object of class c("tidylda").

Examples

# load a document term matrix
data(nih_sample_dtm)

d1 <- nih_sample_dtm[1:50, ]

d2 <- nih_sample_dtm[51:100, ]

# fit a model
m <- tidylda(d1,
  k = 10,
  iterations = 200, burnin = 175
)

# update an existing model by adding documents using old model as prior
m2 <- refit(
  object = m,
  new_data = rbind(d1, d2),
  iterations = 200,
  burnin = 175,
  prior_weight = 1
)

# use an old model to initialize new model and not use old model as prior
m3 <- refit(
  object = m,
  new_data = d2, # new documents only
  iterations = 200,
  burnin = 175,
  prior_weight = NA
)

# add topics while updating a model by adding documents
m4 <- refit(
  object = m,
  new_data = rbind(d1, d2),
  additional_k = 3,
  iterations = 200,
  burnin = 175
)

What this R session can plausibly allocate

Description

Best effort, and advisory only: the value is used to make an error message more informative, never to decide whether to raise one. A limit that moved with the machine would make the same script succeed on one box and fail on another, which is worse than a predictable ceiling.

mem.maxVSize() (base) is finite on macOS, where it is the cap behind "vector memory limit reached", and infinite elsewhere. On Linux the fallback is MemAvailable from /proc/meminfo, which reports the HOST inside a container and so may overstate what is really available. Windows gets neither and is simply omitted from the message.

Usage

session_memory_limit()

Value

a number of bytes, or NA_real_ if nothing could be determined.

Summarize a topic model consistently across methods/functions

Description

Summarizes topics in a model. Called by tidylda and refit.tidylda and used to augment print.tidylda.

Usage

summarize_topics(theta, beta, dtm)

Arguments

Value

Returns a tibble with the following columns: topic is the integer row number of beta. prevalence is the frequency of each topic throughout the corpus it was trained on normalized so that it sums to 100. coherence makes a call to calc_prob_coherence using the default 5 most-probable terms in each topic. top_terms displays the top 5 most-probable terms in each topic.

Note

prevalence should be proportional to P(topic). It is calculated by weighting on document length. So, topics prevalent in longer documents get more weight than topics prevalent in shorter documents. It is calculated by

prevalence <- rowSums(dtm) * theta %>% colSums()

prevalence <- (prevalence * 100) %>% round(3)

An alternative calculation (not implemented here) might have been

prevalence <- colSums(dtm) * t(beta) %>% colSums()

prevalence <- (prevalence * 100) %>% round(3)

Tidy a matrix from a tidylda topic model

Description

Tidy the result of a tidylda topic model

Usage

## S3 method for class 'tidylda'
tidy(x, matrix, log = FALSE, ...)

## S3 method for class 'matrix'
tidy(x, matrix, log = FALSE, ...)

Arguments

Value

Returns a tibble.

If matrix = "beta" then the result is a table of one row per topic and token with the following columns: topic, token, beta

If matrix = "theta" then the result is a table of one row per document and topic with the following columns: document, topic, theta

If matrix = "lambda" then the result is a table of one row per topic and token with the following columns: topic, token, lambda

Functions

• tidy(matrix): Tidy an individual matrix. Useful for predictions and called from tidy.tidylda

Note

If log = TRUE then "log_" will be appended to the name of the third column of the resulting table. e.g "beta" becomes "log_beta".

Examples

dtm <- nih_sample_dtm

lda <- tidylda(data = dtm, k = 10, iterations = 100, burnin = 75)

tidy_beta <- tidy(lda, matrix = "beta")

tidy_theta <- tidy(lda, matrix = "theta")

tidy_lambda <- tidy(lda, matrix = "lambda")

Create a tidy tibble for a dgCMatrix

Description

Create a tidy tibble for a dgCMatrix. Will probably be a PR to tidytext in the future

Usage

tidy_dgcmatrix(x, ...)

Arguments

Value

Returns a triplet matrix with columns "document", "term", and "count"

Utility function to tidy a simple triplet matrix

Description

Utility function to tidy a simple triplet matrix

Usage

tidy_triplet(x, triplets, row_names = NULL, col_names = NULL)

Arguments

Value

returns a triplet matrix in the form of a data frame. The first column indexes rows. The second column indexes columns. The third column contains the i,j values.

Note

This function ported from tidytext, copyright 2017 David Robinson and Julia Silge. Moved the function here for stability reasons, as it is internal to tidytext

Fit a Latent Dirichlet Allocation topic model

Description

Fit a Latent Dirichlet Allocation topic model using warpLDA, a Metropolis-Hastings sampler.

Usage

tidylda(
  data,
  k,
  iterations = NULL,
  burnin = -1,
  alpha = 0.1,
  eta = 0.05,
  optimize_alpha = FALSE,
  calc_likelihood = TRUE,
  calc_r2 = FALSE,
  threads = 1,
  return_data = FALSE,
  verbose = TRUE,
  likelihood_every = 10,
  mh_steps = 1,
  ...
)

Arguments

Details

Fitting uses **warpLDA** (Chen et al., https://arxiv.org/abs/1510.08628), a Metropolis-Hastings sampler that alternates document-ordered and word-ordered passes over the corpus so that each pass touches only a small, cache-resident working set. It replaces the collapsed Gibbs sampler used through version 0.0.7, and is written in Rcpp and parallelized with RcppThread. Some implementation notes follow:

Topic-token and topic-document assignments are not initialized based on a uniform-random sampling, as is common. Instead, topic-token probabilities (i.e. beta) are initialized by sampling from a Dirichlet distribution with eta as its parameter. The same is done for topic-document probabilities (i.e. theta) using alpha. Then an internal function is called (initialize_topic_counts) to run a single sampling iteration to initialize assignments of tokens to topics and topics to documents.

When you use burn-in iterations (i.e. burnin = TRUE), the resulting beta and theta matrices are calculated by averaging over every iteration after the specified number of burn-in iterations. If you do not use burn-in iterations, then the matrices are calculated from the last run only. Ideally, you'd burn in every iteration before convergence, then average over the chain after its converged (and thus every observation is independent).

optimize_alpha is deprecated as of version 0.1.0 and is ignored. It rescaled alpha by topic size each iteration, standing in for fixed-point estimation that was never written. alpha is now fixed for the whole run. The argument will be removed in a future release.

Two log likelihood columns, and they answer different questions. When calc_likelihood = TRUE, the log_likelihood slot has a column of each.

log_likelihood is P(tokens | \theta, \beta): the probability of the observed tokens under the current estimates of theta and beta. It is a plug-in quantity, so it improves monotonically as topics are added and cannot be used to choose k.

log_joint is P(tokens, topics | \alpha, \eta), the collapsed joint, with theta and beta analytically integrated out. This is the quantity most of the LDA literature reports. Integrating out the parameters leaves a discrete distribution, so unlike a density it is always negative, and it carries an implicit penalty for model complexity. It is also the sampler's own target, which makes it the more informative of the two for judging convergence.

Both condition on the current assignment of tokens to topics, so both are within-model diagnostics. Neither is a valid basis for comparing models to each other; that requires P(tokens | \alpha, \eta) with the topic assignments marginalized out, which is intractable and needs the held-out estimators of Wallach et al. (2009).

Both are evaluated every tenth iteration by default rather than every iteration. This is not thinning: the chain advances every iteration and every post-burn-in iteration still contributes to the posterior means. Only these diagnostics, which feed nothing the sampler uses, are computed less often. Pass likelihood_every = 1 to recover a value per iteration. Both accept eta as a scalar, a vector, or a matrix, here and in refit.tidylda and predict.tidylda.

mh_steps sets how many Metropolis-Hastings proposals each token gets per pass. The default of 1 reproduces the reference implementation. Raising it is cheaper than raising iterations by the same factor, because the per-pass overhead — rebuilding the count matrices and the proposal tables — is paid once per iteration however many proposals each token receives. Measured on this package's medium benchmark corpus at k = 100, mh_steps = 4 costs about twice a single-step iteration rather than four times.

That is a statement about cost, not about fit. The proposal distribution is built once per document or word type, so additional steps draw repeatedly from the same proposal, where additional iterations rebuild it from updated counts and alternate the two passes. Whether a short run with several steps matches a long run with one has not been measured; treat mh_steps as an experimental control rather than a drop-in substitute for iterations.

threads sets the number of worker threads. Results are identical at any thread count, so it trades wall clock for cores and nothing else; a model fitted under set.seed() is reproducible whether it was fitted on one thread or twenty. It defaults to 1, so that tidylda never takes cores it was not asked for – which matters when fitting many models inside your own parallel loop.

Value

Returns an S3 object of class tidylda. See new_tidylda.

Examples

# load some data
data(nih_sample_dtm)

# fit a model
set.seed(12345)
m <- tidylda(
  data = nih_sample_dtm[1:20, ], k = 5,
  iterations = 200, burnin = 175
)

str(m)

# predict on held-out documents using Metropolis-Hastings "fold in"
p1 <- predict(m, nih_sample_dtm[21:100, ],
  method = "mh",
  iterations = 200, burnin = 175
)

# predict on held-out documents using the dot product method
p2 <- predict(m, nih_sample_dtm[21:100, ], method = "dot")

# compare the methods
barplot(rbind(p1[1, ], p2[1, ]), beside = TRUE, col = c("red", "blue"))

Bridge function for fitting tidylda topic models

Description

Takes in arguments from various tidylda S3 methods and fits the resulting topic model. Most arguments to this function are documented in tidylda; the two below are specific to the warpLDA engine and reach this function through tidylda's ....

Usage

tidylda_bridge(
  data,
  k,
  iterations,
  burnin,
  alpha,
  eta,
  optimize_alpha,
  calc_likelihood,
  calc_r2,
  threads,
  return_data,
  verbose,
  mc,
  likelihood_every = 10,
  mh_steps = 1,
  ...
)

Arguments

Value

Returns a tidylda S3 object as documented in new_tidylda.