| Type: | Package |
| Title: | Local Large Language Model Inference Engine |
| Version: | 0.7.0 |
| Description: | Enables R users to run large language models locally using 'GGUF' model files and the 'llama.cpp' inference engine. Provides a complete R interface for loading models, generating text completions, extracting vector representations, and streaming responses in real-time. Includes grammar-constrained generation for structured output, text classification, and retrieval-augmented generation (RAG) pipelines. Models can also be served to many users at once as an HTTP API compatible with the 'OpenAI' chat completions API. Supports local inference without requiring cloud APIs or internet connectivity, ensuring complete data privacy and control. Based on the 'llama.cpp' project by Georgi Gerganov (2023) https://github.com/ggml-org/llama.cpp. |
| License: | MIT + file LICENSE |
| URL: | https://github.com/PawanRamaMali/edgemodelr |
| BugReports: | https://github.com/PawanRamaMali/edgemodelr/issues |
| Encoding: | UTF-8 |
| Depends: | R (≥ 4.0) |
| LinkingTo: | Rcpp |
| Imports: | Rcpp (≥ 1.0.0), parallel, utils, tools |
| Suggests: | testthat (≥ 3.1.7), knitr, rmarkdown, curl, shiny, openssl, jsonlite, httr, plumber, promises, later, DBI, RSQLite, callr, ellmer, httpuv |
| SystemRequirements: | GNU make or equivalent for building |
| Note: | Package includes a self-contained 'llama.cpp' implementation for complete functionality without external dependencies. |
| Config/testthat/edition: | 3 |
| RoxygenNote: | 7.3.3 |
| NeedsCompilation: | yes |
| Packaged: | 2026-11-16 15:31:42 UTC; alpine |
| Author: | Pawan Rama Mali [aut, cre, cph], Georgi Gerganov [aut, cph] (Author of llama.cpp and GGML library), The ggml authors [cph] (llama.cpp and GGML contributors), Jeffrey Quesnelle [ctb, cph] (YaRN RoPE implementation), Bowen Peng [ctb, cph] (YaRN RoPE implementation), pi6am [ctb] (DRY sampler from Koboldcpp), Ivan Yurchenko [ctb] (Z-algorithm implementation), Dirk Eddelbuettel [ctb, rev] |
| Maintainer: | Pawan Rama Mali <prm@outlook.in> |
| Repository: | CRAN |
| Date/Publication: | 2026-11-17 07:50:37 UTC |
edgemodelr: Local Large Language Model Inference Engine
Description
Enables R users to run large language models locally using 'GGUF' model files and the 'llama.cpp' inference engine. Provides a complete R interface for loading models, generating text completions, and streaming responses in real-time. Supports local inference without requiring cloud APIs or internet connectivity, ensuring complete data privacy and control. Based on the 'llama.cpp' project by Georgi Gerganov (2023) https://github.com/ggml-org/llama.cpp.
Details
The edgemodelr package provides R bindings for Local Large Language Model Inference Engine using llama.cpp and GGUF model files. This enables completely private, on-device text generation without requiring cloud APIs or internet connectivity.
Main Functions
edge_load_modelLoad a GGUF model file
edge_completionGenerate text completions
edge_stream_completionStream text generation in real-time
edge_chat_streamInteractive chat interface
edge_quick_setupOne-line model download and setup
edge_free_modelRelease model memory
Model Management
edge_list_modelsList available pre-configured models
edge_download_modelDownload models from Hugging Face
Getting Started
Basic usage workflow:
Download a model:
setup <- edge_quick_setup("TinyLlama-1.1B")Generate text:
edge_completion(setup$context, "Hello")Clean up:
edge_free_model(setup$context)
For interactive chat:
setup <- edge_quick_setup("TinyLlama-1.1B")
edge_chat_stream(setup$context)
Examples
See comprehensive examples in the package:
-
system.file("examples/getting_started_example.R", package = "edgemodelr") -
system.file("examples/data_science_assistant_example.R", package = "edgemodelr") -
system.file("examples/text_analysis_example.R", package = "edgemodelr") -
system.file("examples/creative_writing_example.R", package = "edgemodelr") -
system.file("examples/advanced_usage_example.R", package = "edgemodelr")
Run examples:
# Getting started guide
source(system.file("examples/getting_started_example.R", package = "edgemodelr"))
# Data science assistant
source(system.file("examples/data_science_assistant_example.R", package = "edgemodelr"))
System Requirements
C++17 compatible compiler
Sufficient RAM for model size (1GB+ for small models, 8GB+ for 7B models)
GGUF model files (downloaded automatically or manually)
Privacy and Security
This package processes all data locally on your machine. No data is sent to external servers, ensuring complete privacy and control over your text generation workflows.
Author(s)
Pawan Rama Mali prm@outlook.in
See Also
Package repository: https://github.com/PawanRamaMali/edgemodelr
llama.cpp project: https://github.com/ggml-org/llama.cpp
GGUF format: https://github.com/ggml-org/ggml
Clean raw model output into an executable SQL statement
Description
Clean raw model output into an executable SQL statement
Usage
.clean_sql_output(raw)
Compute SHA-256 hash for a file (best-effort, no new hard dependencies)
Description
Compute SHA-256 hash for a file (best-effort, no new hard dependencies)
Usage
.compute_sha256(path)
Arguments
path |
Path to the file |
Value
Lowercase hex string or NA if computation not available
Download using curl command
Description
Download using curl command
Usage
.download_with_curl(
url,
destfile,
hf_token = "",
verbose = TRUE,
max_retries = 3
)
Download using httr package (best for corporate networks)
Description
Download using httr package (best for corporate networks)
Usage
.download_with_httr(url, destfile, hf_token = "", verbose = TRUE)
Download using R's download.file with libcurl
Description
Download using R's download.file with libcurl
Usage
.download_with_r(url, destfile, hf_token = "", verbose = TRUE, max_retries = 3)
Download using wget command
Description
Download using wget command
Usage
.download_with_wget(
url,
destfile,
hf_token = "",
verbose = TRUE,
max_retries = 3
)
Check if a file is a valid GGUF file
Description
Check if a file is a valid GGUF file
Usage
.is_valid_gguf_file(path)
Arguments
path |
Path to the file |
Value
TRUE if valid GGUF, FALSE otherwise
Check directory writeability
Description
Check directory writeability
Usage
.is_writable_dir(path)
Known hashes cache helpers (Trust-On-First-Use optional)
Description
Known hashes cache helpers (Trust-On-First-Use optional)
Usage
.known_hashes_path(cache_dir)
Linearise a record into a "field: value" block
Description
Linearise a record into a "field: value" block
Usage
.linearise_records(data)
Robust file download with retry and resume support
Description
Robust file download with retry and resume support
Usage
.robust_download(url, destfile, verbose = TRUE, max_retries = 3)
Arguments
url |
URL to download |
destfile |
Destination file path |
verbose |
Print progress messages |
max_retries |
Maximum number of retry attempts |
Value
TRUE if successful, FALSE otherwise
Build chat prompt from conversation history
Description
Build a formatted chat prompt from conversation history. When a model context is provided, uses the model's native chat template (e.g., ChatML for Qwen, Llama format for Llama models) for best results.
Usage
build_chat_prompt(history, ctx = NULL)
Arguments
history |
List of conversation turns, each with |
ctx |
Optional model context from |
Value
Formatted prompt string suitable for passing to edge_completion()
Chat with a local model through ellmer
Description
Returns an ellmer chat object backed by a local GGUF model, so the
model works with ellmer's chat, streaming, tool calling and structured
data functions. With model_path, the model is served by
edge_serve in a background R process, which keeps running
until the R session ends and is reused by later calls with the same
model. With base_url, the chat connects to a server you started
yourself.
Usage
chat_edge(
model_path = NULL,
system_prompt = NULL,
base_url = NULL,
n_ctx = 4096L,
n_gpu_layers = 0L,
port = NULL,
params = NULL,
echo = "none",
embeddings = FALSE
)
Arguments
model_path |
Path to a .gguf model file. Either this or
|
system_prompt |
Optional system prompt for the chat. |
base_url |
URL of a running |
n_ctx |
Context size for a server started here (default: 4096). |
n_gpu_layers |
GPU layers for a server started here (default: 0). |
port |
Port for a server started here (default: a free one). |
params |
Optional |
echo |
Passed to |
embeddings |
Also serve |
Details
Tool calling goes through edge_chat_completion: tool
arguments may be strings, numbers, integers, booleans or enums, and the
model makes one call per reply. Structured data works for flat objects
of those types. Streamed replies arrive in one piece when
generation ends (see edge_serve).
Value
An ellmer Chat object.
See Also
Examples
## Not run:
chat <- chat_edge("gemma-3-1b-it-Q4_K_M.gguf",
system_prompt = "Answer in one sentence.")
chat$chat("What is the capital of France?")
# Tool calling with ellmer
get_weather <- ellmer::tool(
function(city) "24 C and sunny",
description = "Get the current weather for a city",
arguments = list(city = ellmer::type_string("City name"))
)
chat$register_tool(get_weather)
chat$chat("What is the weather in Paris?")
## End(Not run)
Build an OpenAI-compatible API for many users
Description
Returns a plumber router that serves a local GGUF model to many users at
once, for Posit Connect or any server that runs plumber APIs. The model
is loaded once, and an edge_engine generates for up to
n_slots requests at the same time with continuous batching; more
requests wait in a queue, so a busy server answers later instead of
failing. Request handlers return promises, so the R process keeps
accepting requests while the engine works.
Usage
edge_api(
model_path,
n_slots = 8L,
n_ctx = 2048L,
n_threads = NULL,
max_queue = 256L,
cache_type = "q8_0",
embeddings = FALSE,
api_key = NULL,
model_name = NULL,
n_gpu_layers = 0L
)
Arguments
model_path |
Path to a .gguf model file. |
n_slots |
Requests generated at once (default: 8). |
n_ctx |
Context per slot, in tokens (default: 2048): the longest prompt plus answer one request may use. |
n_threads |
CPU threads (default: one per core, within the
container's CPU limit). The environment variable
|
max_queue |
Requests that may wait for a slot (default: 256). |
cache_type |
KV cache type for keys and values (default: "q8_0");
see |
embeddings |
FALSE (default), TRUE to also serve
|
api_key |
Optional key. When set, every endpoint but |
model_name |
Name reported as the model (default: the file name). |
n_gpu_layers |
GPU layers (default: 0). |
Details
Endpoints:
-
POST /v1/chat/completionsandPOST /v1/completions: the same requests and replies asedge_serve, including tools,response_formatandstream.usageaddsprompt_tokens_details$cached_tokens, andtimingsgives the milliseconds spent in the queue, reading the prompt and generating, and the tokens per second, as llama.cpp's server does; a stream has them on its last chunk. -
POST /v1/jobs: takes a chat completion request and returns at once (status 202) with a job id.GET /v1/jobs/<id>gives the status, the queue position and the text so far, then the whole chat completion asresponseonce done.DELETE /v1/jobs/<id>cancels it. Jobs suit a Shiny app: it can show the answer as it is written, and no request has to stay open while the model works. Finished jobs are kept for 15 minutes. -
POST /v1/embeddingswhenembeddingsis set. -
GET /v1/models,GET /metrics(engine counters) andGET /health(no key needed).
A full queue answers 503 with Retry-After.
Sizing: each slot gets n_ctx tokens of context, so the KV cache
holds n_slots * n_ctx tokens (at most 32768); cache_type =
"q8_0" halves its memory compared with "f16". On CPU, more slots give
more tokens per second in total until the cores are saturated, typically
at about 8 slots; each request then gets slower. Inside a container the
default thread count follows the container's CPU limit.
Value
A plumber router. Run it with plumber::pr_run(), or return
it from entrypoint.R on Posit Connect. The engine is in
attr(router, "edge_engine").
See Also
Examples
## Not run:
api <- edge_api("gemma-3-1b-it-Q4_K_M.gguf", n_slots = 8)
plumber::pr_run(api, port = 8080)
# entrypoint.R for Posit Connect
edgemodelr::edge_api(Sys.getenv("EDGEMODELR_MODEL"), n_slots = 8,
api_key = Sys.getenv("EDGEMODELR_API_KEY", unset = NA))
## End(Not run)
Ask a question using retrieval-augmented generation
Description
Combines semantic search with text generation: retrieves relevant context from the index, then generates an answer grounded in that context. This is the core RAG (Retrieval-Augmented Generation) function.
Usage
edge_ask(
ctx,
question,
index,
top_k = 3L,
n_predict = 256L,
temperature = 0.3,
system_prompt = NULL,
return_context = FALSE
)
Arguments
ctx |
Model context from edge_load_model() |
question |
The question to answer |
index |
An |
top_k |
Number of context chunks to retrieve (default: 3) |
n_predict |
Maximum tokens for the answer (default: 256) |
temperature |
Sampling temperature (default: 0.3) |
system_prompt |
Optional system-level instruction for the model |
return_context |
If TRUE, return both answer and retrieved context (default: FALSE) |
Value
If return_context = FALSE: a character string with the answer.
If return_context = TRUE: a list with answer, context, and prompt.
See Also
edge_index_documents, edge_search
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
index <- edge_index_documents("./reports/", ctx)
answer <- edge_ask(ctx, "What were Q3 revenues?", index)
cat(answer)
# With context for debugging
result <- edge_ask(ctx, "What were Q3 revenues?", index,
return_context = TRUE)
cat(result$answer)
print(result$context)
## End(Not run)
Performance benchmarking for model inference
Description
Test inference speed and throughput with the current model to measure the effectiveness of optimizations.
Usage
edge_benchmark(
ctx,
prompt = "The quick brown fox",
n_predict = 50,
iterations = 3,
track_memory = FALSE
)
Arguments
ctx |
Model context from edge_load_model() |
prompt |
Test prompt to use for benchmarking (default: standard test) |
n_predict |
Number of tokens to generate for the test |
iterations |
Number of test iterations to average results |
track_memory |
If TRUE, attempt to report peak memory usage (best-effort) |
Value
List with performance metrics
Examples
## Not run:
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
ctx <- setup$context
perf <- edge_benchmark(ctx)
print(perf)
edge_free_model(ctx)
}
## End(Not run)
Cache size information
Description
Cache size information
Usage
edge_cache_info(cache_dir = NULL)
Arguments
cache_dir |
Cache directory path |
Value
List with total_size_mb and file_count
Generate a chat completion using the model's native template
Description
Formats messages using the model's built-in chat template (read from GGUF metadata), generates a completion, and returns only the assistant's response. This is the recommended way to do multi-turn chat, as it uses the correct template format for the specific model (e.g., ChatML for Qwen, Llama format for Llama models).
Usage
edge_chat_completion(
ctx,
messages,
n_predict = 256L,
temperature = 0.7,
top_p = 0.95,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL,
tools = NULL,
tool_choice = "auto"
)
Arguments
ctx |
Model context from edge_load_model() |
messages |
A list of message objects, each with |
n_predict |
Maximum tokens to generate (default: 256) |
temperature |
Sampling temperature (default: 0.7) |
top_p |
Nucleus sampling threshold (default: 0.95) |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
tools |
Optional list of tools from |
tool_choice |
With |
Details
With tools, the tools are described in the system message and
generation is constrained by a grammar, so a tool call is always valid
JSON naming a known tool, with every argument present and of the right
type. The model writes a call as
{"tool": "name", "arguments": {...}}. Run the tool yourself, append
result$message and a message with role = "tool", the tool's
name and its result as content, then call again so the model
can use the result. Tool results are passed to the model as user
messages, since not every chat template has a tool role.
The tool call is always generated greedily; temperature and the
other sampling arguments apply to the text answer.
Value
Without tools, a character string containing only the
assistant's response text. With tools, a list with
content (the text answer, or NULL when the model called a tool),
tool_calls (a list of calls, each with name and
arguments, a named list) and message (the assistant
message to append to messages before the next call).
See Also
build_chat_prompt, edge_completion,
edge_tool
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Simple chat
answer <- edge_chat_completion(ctx, list(
list(role = "user", content = "What is R?")
))
cat(answer)
# Multi-turn with system prompt
answer <- edge_chat_completion(ctx, list(
list(role = "system", content = "You are a helpful R tutor."),
list(role = "user", content = "What is a data.frame?"),
list(role = "assistant", content = "A data.frame is a table..."),
list(role = "user", content = "How do I filter rows?")
))
# Tool calling
weather <- edge_tool("get_weather", "Get the current weather for a city",
list(city = "string", unit = c("celsius", "fahrenheit")))
msgs <- list(list(role = "user", content = "Is it warm in Paris?"))
res <- edge_chat_completion(ctx, msgs, tools = list(weather), temperature = 0)
if (length(res$tool_calls) > 0) {
call <- res$tool_calls[[1]]
call$arguments$city # "Paris"
msgs <- c(msgs, list(res$message,
list(role = "tool", name = call$name, content = "24 C, sunny")))
res <- edge_chat_completion(ctx, msgs, tools = list(weather), temperature = 0)
cat(res$content)
}
edge_free_model(ctx)
## End(Not run)
Interactive chat session with streaming responses
Description
Interactive chat session with streaming responses
Usage
edge_chat_stream(
ctx,
system_prompt = NULL,
max_history = 10,
n_predict = 200L,
temperature = 0.8,
verbose = TRUE,
top_p = 0.95,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL
)
Arguments
ctx |
Model context from edge_load_model() |
system_prompt |
Optional system prompt to set context |
max_history |
Maximum conversation turns to keep in context (default: 10) |
n_predict |
Maximum tokens per response (default: 200) |
temperature |
Sampling temperature (default: 0.8) |
verbose |
Whether to print status messages (default: TRUE) |
top_p |
Top-p sampling parameter (default: 0.95) |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
Value
NULL (runs interactively)
Examples
## Not run:
setup <- edge_quick_setup("TinyLlama-1.1B")
ctx <- setup$context
if (!is.null(ctx)) {
# Start interactive chat with streaming
# edge_chat_stream(ctx,
# system_prompt = "You are a helpful R programming assistant.")
edge_free_model(ctx)
}
## End(Not run)
Classify text into predefined categories
Description
Classify text into predefined categories using grammar-constrained generation. The grammar constraint ensures the model can only output one of the provided categories.
Usage
edge_classify(
ctx,
text,
categories,
instruction = NULL,
temperature = 0.1
)
Arguments
ctx |
Model context from edge_load_model() |
text |
Text to classify (character string or vector for batch) |
categories |
Character vector of allowed categories |
instruction |
Optional classification instruction |
temperature |
Sampling temperature (default: 0.1) |
Value
Character string (single text) or character vector (batch) with the predicted category. Output is guaranteed to be one of the specified categories.
See Also
edge_extract, edge_grammar_completion
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Single classification
result <- edge_classify(ctx,
"I love this product!",
categories = c("positive", "negative", "neutral"))
# Batch classification
texts <- c("Great product!", "Terrible experience", "It was okay")
results <- edge_classify(ctx, texts,
categories = c("positive", "negative", "neutral"))
edge_free_model(ctx)
## End(Not run)
Clean up cache directory and manage storage
Description
Remove outdated model files from the cache directory to comply with CRAN
policies about actively managing cached content and keeping sizes small.
The CUDA backend in the cuda subdirectory and the local checksum store
are never removed.
Usage
edge_clean_cache(
cache_dir = NULL,
max_age_days = getOption("edgemodelr.cache_max_age_days", 30),
max_size_mb = getOption("edgemodelr.cache_max_size_mb", 5000),
use_lru = TRUE,
ask = TRUE,
verbose = TRUE
)
Arguments
cache_dir |
Cache directory path (default: user cache directory) |
max_age_days |
Maximum age of files to keep in days (default: option edgemodelr.cache_max_age_days or 30) |
max_size_mb |
Maximum total cache size in MB (default: option edgemodelr.cache_max_size_mb or 5000) |
use_lru |
If TRUE, evict least-recently-used files when size exceeds limit |
ask |
Whether to ask for user confirmation before deletion (only in interactive sessions) |
verbose |
Whether to print status messages (default: TRUE) |
Value
Invisible list of deleted files
Examples
## Not run:
# Clean cache files older than 30 days
edge_clean_cache()
# Clean cache with custom settings
edge_clean_cache(max_age_days = 7, max_size_mb = 100)
## End(Not run)
Generate text completion using loaded model
Description
Generate text completion using loaded model
Usage
edge_completion(
ctx,
prompt,
n_predict = 128L,
temperature = 0.8,
top_p = 0.95,
timeout_seconds = NULL,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL
)
Arguments
ctx |
Model context from edge_load_model() |
prompt |
Input text prompt |
n_predict |
Maximum tokens to generate (default: 128) |
temperature |
Sampling temperature (default: 0.8). 0 gives greedy decoding: the same prompt always gives the same text. |
top_p |
Top-p sampling parameter (default: 0.95) |
timeout_seconds |
Optional timeout in seconds for inference |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
Value
Generated text as character string
Examples
## Not run:
# Basic completion example
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
ctx <- setup$context
# Simple completion
result <- edge_completion(ctx, "The capital of France is", n_predict = 50)
cat("Result:", result, "\n")
# Completion with custom parameters
creative_result <- edge_completion(
ctx,
"Write a short poem about data science:",
n_predict = 100,
temperature = 0.9,
top_p = 0.8
)
cat("Creative result:", creative_result, "\n")
# Reproducible sampling, less repetition, stop at the first blank line
edge_completion(ctx, "List three colours:", seed = 42,
repeat_penalty = 1.2, stop = "\n\n")
edge_free_model(ctx)
}
## End(Not run)
Return the default path for the installed CUDA backend DLL
Description
Return the default path for the installed CUDA backend DLL
Usage
edge_cuda_default_path()
Check whether a CUDA backend is installed and active
Description
Reports the installation and activation status of the GPU (CUDA) backend.
Use this to verify that GPU inference is available before loading a model
with n_gpu_layers > 0.
Usage
edge_cuda_info()
Value
A named list with three elements:
- installed
Logical.
TRUEif a CUDA backend DLL is present in the edgemodelr cache directory.- active
Logical.
TRUEif the CUDA backend has been loaded into the current R session (i.e.edge_reload_cuda()was called).- path
Character. Path to the active CUDA DLL, the installed DLL (if not yet active), or
NAif no backend is found.
See Also
edge_install_cuda, edge_install_cuda_toolkit,
edge_reload_cuda
Examples
## Not run:
info <- edge_cuda_info()
if (info$active) {
cat("GPU inference active:", info$path, "\n")
} else if (info$installed) {
cat("CUDA backend installed but not active. Call edge_reload_cuda().\n")
} else {
cat("No CUDA backend. Run edge_install_cuda() to enable GPU support.\n")
}
## End(Not run)
Download a GGUF model from Hugging Face
Description
Download a GGUF model from Hugging Face
Usage
edge_download_model(
model_id,
filename = NULL,
cache_dir = NULL,
force_download = FALSE,
verify_checksum = TRUE,
expected_sha256 = NULL,
trust_first_use = FALSE,
verbose = TRUE
)
Arguments
model_id |
Hugging Face model identifier (e.g., "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"),
or a friendly model name from |
filename |
Specific GGUF file to download. Optional when using a friendly model name. |
cache_dir |
Directory to store downloaded models (default: "~/.cache/edgemodelr") |
force_download |
Force re-download even if file exists |
verify_checksum |
Verify SHA-256 checksum if available (default: TRUE) |
expected_sha256 |
Optional expected SHA-256 hash for the model file |
trust_first_use |
Store a local hash if no known hash exists (default: FALSE) |
verbose |
Whether to print download progress messages |
Value
Path to the downloaded model file
Examples
## Not run:
# Download using friendly name (recommended)
model_path <- edge_download_model("Qwen3-0.6B")
model_path <- edge_download_model("mistral-7b")
# Download using HuggingFace repo ID
model_path <- edge_download_model(
model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
filename = "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
)
## End(Not run)
Download a model from a direct URL
Description
Downloads a GGUF model file from any URL. Supports resume and validates GGUF format. This function is useful for downloading models from GPT4All CDN or other direct sources that don't require authentication.
Usage
edge_download_url(
url,
filename,
cache_dir = NULL,
force_download = FALSE,
verify_checksum = TRUE,
expected_sha256 = NULL,
trust_first_use = FALSE,
verbose = TRUE
)
Arguments
url |
Direct download URL for the model |
filename |
Local filename to save as |
cache_dir |
Directory to store downloaded models (default: user cache directory) |
force_download |
Force re-download even if file exists |
verify_checksum |
Verify SHA-256 checksum if available (default: TRUE) |
expected_sha256 |
Optional expected SHA-256 hash for the file |
trust_first_use |
Store a local hash if no known hash exists (default: FALSE) |
verbose |
Whether to print progress messages |
Value
Path to the downloaded model file
Examples
## Not run:
# Download from GPT4All CDN (large file, not run in checks)
model_path <- edge_download_url(
url = "https://gpt4all.io/models/gguf/mistral-7b-instruct-v0.1.Q4_0.gguf",
filename = "mistral-7b.gguf"
)
## End(Not run)
Extract text embeddings from a model
Description
Computes dense vector embeddings for one or more text inputs using the loaded model. These embeddings can be used for semantic search, clustering, similarity comparison, and as input to downstream models.
Usage
edge_embeddings(ctx, texts, normalize = TRUE)
Arguments
ctx |
Model context from edge_load_model() |
texts |
Character vector of texts to embed |
normalize |
Whether to L2-normalize the embeddings (default: TRUE). Normalized embeddings allow cosine similarity to be computed as a simple dot product. |
Details
For best results, use a model designed for embeddings (e.g., nomic-embed-text). However, generative models can also produce useful embeddings from their hidden states.
The embedding dimension depends on the model architecture. Use
edge_model_n_embd() to query the dimension.
Value
A numeric matrix with dimensions (n_texts x n_embd), where each row is the embedding vector for the corresponding input text.
See Also
edge_similarity, edge_similarity_matrix, edge_model_n_embd
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Single embedding
emb <- edge_embeddings(ctx, "The cat sat on the mat")
dim(emb) # 1 x n_embd
# Multiple embeddings
texts <- c("cats are great", "dogs are loyal", "the stock market crashed")
embs <- edge_embeddings(ctx, texts)
# Compute similarity
edge_similarity(embs[1,], embs[2,]) # high (both about pets)
edge_similarity(embs[1,], embs[3,]) # low (different topics)
edge_free_model(ctx)
## End(Not run)
Run many requests at once on one model
Description
Starts a serving engine on a loaded model: a background thread that
generates for several requests at the same time with continuous batching,
the way llama.cpp's own server does. Each of n_slots slots holds
one request; every step decodes the next token of each running request
in one batch, so on CPU total throughput grows with the number of slots
while each request gets a little slower. Requests beyond the slots wait in
a queue, first come first served. A slot keeps its tokens after a request
ends, and a later request that starts the same way (the same system
prompt, or the next turn of a chat) reuses them instead of reading them
again.
Usage
edge_engine(ctx, n_slots = NULL, max_queue = 256L)
edge_engine_submit(
engine,
prompt,
n_predict = 256L,
temperature = 0.7,
top_p = 0.95,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL,
grammar = NULL
)
edge_engine_poll(engine, ids)
edge_engine_release(engine, ids)
edge_engine_cancel(engine, ids)
edge_engine_result(engine, id)
edge_engine_generate(
engine,
prompts,
n_predict = 256L,
temperature = 0.7,
top_p = 0.95,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL,
grammar = NULL,
timeout = Inf
)
edge_engine_stats(engine)
edge_engine_stop(engine)
Arguments
ctx |
A model loaded with |
n_slots |
Requests generated at once (default: the model's
|
max_queue |
Requests that may wait (default: 256). A submit beyond
it fails with an error of class |
engine |
An engine from |
prompt, prompts |
Prompt text, already formatted (see
|
n_predict |
Most tokens to generate (default: 256). |
temperature, top_p, top_k, min_p, repeat_penalty, seed, stop |
Sampling,
as in |
grammar |
Optional GBNF grammar with a |
ids, id |
Request ids from |
timeout |
Seconds |
Details
The engine runs beside R: edge_engine_submit() returns at once
with a request id, and R stays free while the request is generated.
edge_engine_poll() reports progress and the text so far. A
finished request is kept until edge_engine_release() frees it
(an unfinished one is cancelled first), so a client that polls should
release each request once it has read the result.
edge_engine_result() gives a promise of the finished request (for
plumber and Shiny) and edge_engine_generate() submits prompts and
waits for all of them; both free the requests themselves.
While the engine runs, the model belongs to it: the direct completion
functions refuse to use ctx until edge_engine_stop().
Tokenizing and chat templates still work. edge_api builds
an HTTP API on an engine. An internal llama.cpp error on the engine's
thread cannot become an R error and ends the R process, as it would end
llama.cpp's own server; on Posit Connect the process is then restarted.
Value
edge_engine(): an edge_engine object.
edge_engine_submit(): an integer request id.
edge_engine_poll() and edge_engine_generate(): a data
frame with one row per request: id, status ("queued",
"running", "done" or "unknown"), text (so far, for an
unfinished request), finish ("stop", "length", "cancelled" or
"error"), error, prompt_tokens,
completion_tokens, cached_tokens (prompt tokens reused
from a slot), queue_position, queue_seconds,
first_token_seconds and total_seconds.
edge_engine_result(): a promise of a list with the same fields,
rejected with an error of class edge_cancelled if the request is
cancelled.
edge_engine_stats(): a list of counters.
See Also
Examples
## Not run:
ctx <- edge_load_model("gemma-3-1b-it-Q4_K_M.gguf", n_ctx = 8192, n_parallel = 4)
engine <- edge_engine(ctx)
prompts <- vapply(c("Rome", "Paris", "Oslo", "Lima"), function(city) {
build_chat_prompt(list(list(role = "user",
content = paste("One fact about", city))), ctx = ctx)
}, character(1))
out <- edge_engine_generate(engine, prompts, n_predict = 60, temperature = 0)
out$text
edge_engine_stats(engine)
edge_engine_stop(engine)
## End(Not run)
Extract structured data from text
Description
High-level function that combines prompt construction with grammar-constrained generation to extract structured data from text. Returns a parsed R list.
Usage
edge_extract(
ctx,
text,
schema,
instruction = NULL,
n_predict = 512L,
temperature = 0.2,
draft = FALSE
)
Arguments
ctx |
Model context from edge_load_model() |
text |
The input text to analyze |
schema |
A named list defining the extraction schema (see |
instruction |
Optional instruction to guide extraction (default: auto-generated) |
n_predict |
Maximum tokens to generate (default: 512) |
temperature |
Sampling temperature (default: 0.2, very low for factual extraction) |
draft |
If TRUE, first let the model answer in free text, then run the grammar-constrained pass with that draft in the prompt (default: FALSE). It adds a second generation pass, so it takes two to four times as long. |
Details
The grammar guarantees well-formed JSON, which is where most of its accuracy
gain comes from. Forcing a very small model straight into a strict format
can, however, cost answer accuracy. With draft = TRUE the model first
reasons about the fields without constraints and the grammar is applied only
to the final answer. Published results for this "draft, then constrain"
pattern show gains for 4B-14B models (e.g. micro-F1 0.644 to 0.795 for
Qwen3-14B on GenSIE 2026 zero-shot extraction). It is most worth trying with
models under about 4B parameters, or when plain extraction leaves fields wrong.
Value
A named list with the extracted fields, or the raw JSON string if parsing fails. Requires the jsonlite package for automatic parsing.
See Also
edge_json_grammar, edge_grammar_completion,
edge_classify, edge_extract_batch
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
result <- edge_extract(ctx,
"I absolutely love this product! Best purchase ever.",
schema = list(
sentiment = c("positive", "negative", "neutral"),
confidence = "number"
))
# result$sentiment == "positive"
# result$confidence == 0.95
edge_free_model(ctx)
## End(Not run)
Extract structured data from multiple texts
Description
Batch version of edge_extract that processes a vector of texts
and returns a data frame.
Usage
edge_extract_batch(
ctx,
texts,
schema,
instruction = NULL,
n_predict = 512L,
temperature = 0.2,
progress = TRUE,
draft = FALSE
)
Arguments
ctx |
Model context from edge_load_model() |
texts |
Character vector of texts to process |
schema |
A named list defining the extraction schema (see |
instruction |
Optional instruction to guide extraction |
n_predict |
Maximum tokens to generate per text (default: 512) |
temperature |
Sampling temperature (default: 0.2) |
progress |
Show progress messages (default: TRUE) |
draft |
Passed to |
Value
A data frame with one row per text and columns for each schema field
See Also
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
reviews <- c("Love it!", "Worst purchase ever", "It's okay I guess")
results <- edge_extract_batch(ctx, reviews,
schema = list(
sentiment = c("positive", "negative", "neutral"),
confidence = "number"
))
# results is a data frame:
# sentiment confidence
# 1 positive 0.95
# 2 negative 0.90
# 3 neutral 0.60
edge_free_model(ctx)
## End(Not run)
Find and prepare GGUF models for use with edgemodelr
Description
This function finds compatible GGUF model files from various sources including Ollama installations, custom directories, or any folder containing GGUF files. It tests each model for compatibility with edgemodelr and creates organized copies or links for easy access.
Usage
edge_find_gguf_models(
source_dirs = NULL,
target_dir = NULL,
create_links = TRUE,
model_pattern = NULL,
test_compatibility = TRUE,
min_size_mb = 50,
verbose = TRUE
)
Arguments
source_dirs |
Vector of directories to search for GGUF files. If NULL, automatically searches common locations including Ollama installation. |
target_dir |
Directory where to create links/copies of compatible models. If NULL, creates a "local_models" directory in the current working directory. |
create_links |
Logical. If TRUE (default), creates symbolic links to save disk space. If FALSE, copies the files (uses more disk space but more compatible). |
model_pattern |
Optional pattern to filter model files by name. |
test_compatibility |
Logical. If TRUE (default), tests each GGUF file for compatibility with edgemodelr before including it. |
min_size_mb |
Minimum file size in MB to consider (default: 50MB). Helps filter out config files and focus on actual models. |
verbose |
Logical. Whether to print detailed progress information. |
Details
This function performs the following steps:
Searches specified directories (or auto-detects common locations)
Identifies GGUF format files above the minimum size threshold
Optionally tests each file for compatibility with edgemodelr
Creates organized symbolic links or copies in the target directory
Returns detailed information about working models
The function automatically searches these locations if no source_dirs specified:
Ollama models directory (~/.ollama/models or %USERPROFILE%/.ollama/models)
Current working directory
~/models directory (if exists)
Common model storage locations
Value
List containing information about compatible models, including paths and metadata
Examples
## Not run:
# Basic usage - auto-detect and test all GGUF models
models_info <- edge_find_gguf_models()
if (!is.null(models_info) && length(models_info$models) > 0) {
# Load the first compatible model
ctx <- edge_load_model(models_info$models[[1]]$path)
result <- edge_completion(ctx, "Hello", n_predict = 20)
edge_free_model(ctx)
}
# Search specific directories
models_info <- edge_find_gguf_models(source_dirs = c("~/Downloads", "~/models"))
# Skip compatibility testing (faster but less reliable)
models_info <- edge_find_gguf_models(test_compatibility = FALSE)
# Copy files instead of creating links
models_info <- edge_find_gguf_models(create_links = FALSE)
# Filter for specific models
models_info <- edge_find_gguf_models(model_pattern = "llama")
## End(Not run)
Find and load Ollama models
Description
Utility functions to discover and work with locally stored Ollama models. Ollama stores models as SHA-256 named blobs which are GGUF files that can be used directly with edgemodelr.
Usage
edge_find_ollama_models(
ollama_dir = NULL,
test_compatibility = FALSE,
max_size_gb = 10
)
Arguments
ollama_dir |
Optional path to Ollama models directory. If NULL, will auto-detect. |
test_compatibility |
If TRUE, test if each model can be loaded successfully |
max_size_gb |
Maximum model size in GB to consider (default: 10) |
Value
List with ollama_path and discovered models information
Examples
## Not run:
# Find Ollama models
ollama_info <- edge_find_ollama_models()
if (!is.null(ollama_info) && length(ollama_info$models) > 0) {
# Use first compatible model
model_path <- ollama_info$models[[1]]$path
ctx <- edge_load_model(model_path)
result <- edge_completion(ctx, "Hello", n_predict = 10)
edge_free_model(ctx)
}
## End(Not run)
Free model context and release memory
Description
Free model context and release memory
Usage
edge_free_model(ctx)
Arguments
ctx |
Model context from edge_load_model() |
Value
NULL (invisibly)
Examples
## Not run:
# Requires a downloaded model (not run in checks)
model_path <- "model.gguf"
if (file.exists(model_path)) {
ctx <- edge_load_model(model_path)
# ... use model ...
edge_free_model(ctx) # Clean up
}
## End(Not run)
Generate text constrained by a GBNF grammar
Description
Uses llama.cpp's grammar-constrained sampling to force output to conform to a GBNF grammar specification. This ensures structured, parseable output.
Usage
edge_grammar_completion(
ctx,
prompt,
grammar,
grammar_root = "root",
n_predict = 512L,
temperature = 0.3,
top_p = 0.95,
jump_forward = TRUE,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL
)
Arguments
ctx |
Model context from edge_load_model() |
prompt |
The input prompt |
grammar |
A GBNF grammar string defining allowed output structure |
grammar_root |
The root rule name in the grammar (default: "root") |
n_predict |
Maximum tokens to generate (default: 512) |
temperature |
Sampling temperature (default: 0.3, lower for structured output). 0 gives greedy decoding: the same prompt always gives the same text. |
top_p |
Nucleus sampling threshold (default: 0.95) |
jump_forward |
If TRUE (default), text that the grammar allows in only
one way (JSON keys, separators, the rest of an enum value once it is
unambiguous) is added without sampling, and its tokens are decoded in one
batch, in fewer decode steps. The forced text is tokenized the standard way,
which can differ from how the model would split it, so the output can
differ slightly from |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
Details
GBNF (GGML BNF) is a format for defining formal grammars that constrain model output. This is useful for generating JSON, XML, or any structured format.
Jump-forward leaves the last token of each forced stretch for the model to
sample, since that token may merge with the text after it (a model writes
an opening brace and quote as one token). It saves the most with long
JSON keys and edge_json_grammar(compact = TRUE), where the layout
between fields is fixed and so is forced too: with four long keys a 0.6B
model wrote the object about 3x faster. With short keys the saving is small.
Common GBNF patterns:
JSON object: Use
edge_json_grammar()for convenienceEnum/choices:
'root ::= "yes" | "no" | "maybe"'Number:
'root ::= [0-9]+'
Value
Character string containing only the generated text (not the prompt)
See Also
edge_json_grammar, edge_extract, edge_classify
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Force yes/no output
grammar <- 'root ::= "yes" | "no"'
result <- edge_grammar_completion(ctx, "Is the sky blue? Answer:", grammar)
# Force JSON output
json_grammar <- edge_json_grammar(list(
sentiment = c("positive", "negative", "neutral"),
confidence = "number"
))
result <- edge_grammar_completion(ctx,
"Analyze sentiment: 'I love this product'\nJSON:",
json_grammar)
edge_free_model(ctx)
## End(Not run)
Build an embedding index from text documents
Description
Reads text files from a directory (or accepts text directly), splits into chunks, computes embeddings, and returns an index object for semantic search and retrieval-augmented generation.
Usage
edge_index_documents(
source,
ctx,
chunk_size = 500L,
chunk_overlap = 50L,
file_pattern = "*.txt",
normalize = TRUE,
progress = TRUE
)
Arguments
source |
Either a directory path containing text files, or a character vector of text chunks to index directly. |
ctx |
Model context from edge_load_model() (used for embedding) |
chunk_size |
Approximate number of characters per chunk (default: 500) |
chunk_overlap |
Number of characters of overlap between chunks (default: 50) |
file_pattern |
Glob pattern for files to read (default: "*.txt") |
normalize |
Normalize embeddings (default: TRUE) |
progress |
Show progress messages (default: TRUE) |
Value
An edge_index object containing chunks, embeddings, and source metadata.
See Also
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Index a directory
index <- edge_index_documents("./reports/", ctx)
# Or index text directly
index <- edge_index_documents(
c("Revenue grew 15%", "New product launched"), ctx)
# Search the index
results <- edge_search(index, ctx, "revenue growth")
## End(Not run)
Install the CUDA backend for GPU-accelerated inference
Description
Downloads a pre-built GGML CUDA backend shared library and stores it in the
edgemodelr cache. After installation, restart your R session (or call
edge_reload_cuda()) so the GPU backend is picked up during model loading.
On Windows, the CUDA runtime libraries (cudart, cublas) must also be
present. Run edge_install_cuda_toolkit() once after this function
to install them automatically.
Usage
edge_install_cuda(cuda_version = "13.4", force = FALSE, llama_build = "b11960")
Arguments
cuda_version |
CUDA version string. One of |
force |
Re-download even if already installed. |
llama_build |
llama.cpp build number to pull the backend from
(default: |
Value
Invisibly returns the path to the installed CUDA backend DLL.
See Also
edge_install_cuda_toolkit, edge_reload_cuda,
edge_cuda_info
Examples
## Not run:
edge_install_cuda() # GPU backend installed
edge_install_cuda_toolkit() # Install CUDA runtime DLLs (Windows)
ctx <- edge_load_model("model.gguf", n_gpu_layers = 35)
## End(Not run)
Install CUDA runtime libraries required for GPU inference
Description
The CUDA backend (ggml-cuda-XX.dll) requires two sets of runtime DLLs
that are not included with the NVIDIA display driver:
-
nvcudart_hybrid64.dll: CUDA hybrid runtime, already present on your system in the Windows DriverStore (installed with the GPU driver). This function copies it to the edgemodelr cache.
-
cublas64_13.dll / cublasLt64_13.dll: cuBLAS linear-algebra library (~400 MB download from NVIDIA's official redistrib server).
After running this function, call edge_reload_cuda() and load a model
with n_gpu_layers = -1 to run on your GPU.
This function is only needed on Windows. On Linux and macOS, CUDA support
is enabled at build time via the EDGEMODELR_CUDA=1 environment variable.
Usage
edge_install_cuda_toolkit(cuda_version = "13.4", force = FALSE)
Arguments
cuda_version |
CUDA major version string, e.g. |
force |
Reinstall even if the runtime DLLs are already present. |
Value
Invisibly returns the cuda cache directory path.
See Also
edge_install_cuda, edge_reload_cuda,
edge_cuda_info
Examples
## Not run:
edge_install_cuda() # install ggml-cuda GPU backend DLL (~140 MB)
edge_install_cuda_toolkit() # install CUDA runtime DLLs (~400 MB, one-time)
edge_reload_cuda() # activate in this R session
ctx <- edge_load_model("model.gguf", n_gpu_layers = -1)
result <- edge_completion(ctx, "Hello", n_predict = 20)
## End(Not run)
Generate a GBNF grammar for JSON output from a schema
Description
Converts a simple R list schema into a GBNF grammar string that constrains model output to valid JSON matching the schema.
Usage
edge_json_grammar(schema, compact = FALSE)
Arguments
schema |
A named list where each element defines a field. Values can be:
|
compact |
If TRUE, the layout is fixed to |
Value
A GBNF grammar string suitable for use with edge_grammar_completion
See Also
edge_grammar_completion, edge_extract
Examples
# Schema with enum and free-text fields
grammar <- edge_json_grammar(list(
sentiment = c("positive", "negative", "neutral"),
confidence = "number",
explanation = "string"
))
cat(grammar)
List popular pre-configured models
Description
List popular pre-configured models
Usage
edge_list_models()
Value
Data frame with model information
Load a local GGUF model for inference
Description
Load a local GGUF model for inference
Usage
edge_load_model(
model_path,
n_ctx = 2048L,
n_gpu_layers = 0L,
n_threads = NULL,
flash_attn = TRUE,
embeddings = FALSE,
prompt_cache = FALSE,
lora = NULL,
lora_scale = 1,
n_parallel = 1L,
cache_type_k = "f16",
cache_type_v = "f16"
)
Arguments
model_path |
Path to a .gguf model file |
n_ctx |
Maximum context length (default: 2048) |
n_gpu_layers |
Number of layers to offload to GPU (default: 0, CPU-only). Use -1 to offload all layers (requires CUDA backend via edge_install_cuda()). |
n_threads |
Number of CPU threads for inference (default: NULL = one per physical core). Extra hyperthreads slow generation down, often by half. Set to a lower value to leave cores free for other tasks. |
flash_attn |
Enable flash attention for faster inference (default: TRUE). Reduces memory usage and improves speed. Set to FALSE for maximum compatibility. |
embeddings |
Enable embedding extraction mode (default: FALSE).
Must be TRUE to use |
prompt_cache |
Keep the KV cache between calls and reuse the part of a new prompt that matches the start of the previous one, so only the rest is processed (default: FALSE). Much faster when prompts share a long fixed beginning, such as a system prompt. Each completion still depends only on its own prompt. |
lora |
Path(s) to LoRA adapters in GGUF format to apply on top of the
model (default: NULL, none). The adapters must be made for this base
model. A PEFT adapter can be converted with llama.cpp's
|
lora_scale |
Scale for each adapter, recycled to the number of
adapters (default: 1). 0 turns an adapter off. Change it later with
|
n_parallel |
How many prompts |
cache_type_k, cache_type_v |
Number format of the keys and values in
the KV cache (default: "f16"). One of "f16", "bf16", "q8_0", "q5_1",
"q5_0", "q4_1", "q4_0" or "iq4_nl". "q8_0" for both halves the memory
the cache takes, with almost no change in output, so a larger
|
Value
External pointer to the loaded model context
Examples
## Not run:
# Quick setup with automatic model download (downloads ~700MB)
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
ctx <- setup$context
# Generate completion
result <- edge_completion(ctx, "Explain R data.frame:", n_predict = 100)
cat(result)
# Free model when done
edge_free_model(ctx)
}
# Manual model loading with threading control
model_path <- "path/to/your/model.gguf"
if (file.exists(model_path)) {
ctx <- edge_load_model(model_path, n_ctx = 2048, n_threads = 4, flash_attn = TRUE)
# ... use model ...
edge_free_model(ctx)
}
## End(Not run)
Load an Ollama model by partial SHA-256 hash
Description
Find and load an Ollama model using a partial SHA-256 hash instead of the full path. This is more convenient than typing out the full blob path.
Usage
edge_load_ollama_model(partial_hash, n_ctx = 2048L, n_gpu_layers = 0L)
Arguments
partial_hash |
First few characters of the SHA-256 hash |
n_ctx |
Maximum context length (default: 2048) |
n_gpu_layers |
Number of layers to offload to GPU (default: 0) |
Value
Model context if successful, throws error if not found or incompatible
Examples
## Not run:
# Load model using first 8 characters of SHA hash
# ctx <- edge_load_ollama_model("b112e727")
# result <- edge_completion(ctx, "Hello", n_predict = 10)
# edge_free_model(ctx)
## End(Not run)
Get or change the scale of loaded LoRA adapters
Description
Adapters are loaded with edge_load_model(lora = ...). Changing a
scale takes effect on the next call and needs no reload, so one base model
can switch between adapters, or turn them off with 0, from call to call.
Usage
edge_lora_scale(ctx, scale = NULL)
Arguments
ctx |
Model context from |
scale |
NULL to only read the scales, or one number per loaded
adapter, in the order they were given to |
Details
Changing a scale clears the context's KV cache, so with
prompt_cache = TRUE the next prompt is processed in full.
Value
The adapter scales, a numeric vector (invisibly when setting)
Examples
## Not run:
ctx <- edge_load_model("base.gguf", lora = c("terms.gguf", "labels.gguf"),
lora_scale = c(1, 0))
edge_completion(ctx, "Map 'Once daily' to a CDISC term:")
edge_lora_scale(ctx, c(0, 1)) # switch to the second adapter
edge_lora_scale(ctx) # 0 1
edge_free_model(ctx)
## End(Not run)
Apply a prompt template to a vector of texts
Description
Maps a prompt template over a character vector, generating completions for each element. This is the primary function for batch LLM operations on data frames. A progress indicator is printed to the console.
Usage
edge_map(
ctx,
texts,
prompt_template,
n_predict = 128L,
temperature = 0.7,
top_p = 0.95,
grammar = NULL,
progress = TRUE,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL
)
Arguments
ctx |
Model context from edge_load_model() |
texts |
Character vector of input texts |
prompt_template |
A string with |
n_predict |
Maximum tokens to generate per text (default: 128) |
temperature |
Sampling temperature (default: 0.7) |
top_p |
Nucleus sampling threshold (default: 0.95) |
grammar |
Optional GBNF grammar string to constrain output |
progress |
Show progress messages (default: TRUE) |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
Value
Character vector of completions, same length as texts
See Also
edge_classify, edge_extract_batch
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
# Summarize a column
df <- data.frame(review = c("Great product!", "Terrible, broke on day 1"))
df$summary <- edge_map(ctx, df$review,
"Summarize in 5 words: {text}")
# With a custom prompt function
df$analysis <- edge_map(ctx, df$review, function(text) {
paste0("Product review: ", text, "\nOne-word sentiment:")
})
edge_free_model(ctx)
## End(Not run)
Get the embedding dimension of a loaded model
Description
Returns the size of the embedding vectors produced by the loaded model.
Usage
edge_model_n_embd(ctx)
Arguments
ctx |
Model context from edge_load_model() |
Value
Integer giving the embedding vector dimension
See Also
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
edge_model_n_embd(ctx) # e.g., 4096
edge_free_model(ctx)
## End(Not run)
Generate an English narrative from structured data
Description
Turns a named list, single-row data.frame, or multi-row data.frame into one or more short English sentences describing the data. The function linearises the input as labelled "field: value" lines and asks the model to narrate them without inferring anything that is not in the input.
Usage
edge_narrate(
ctx,
data,
instruction = NULL,
max_words = 60L,
n_predict = 200L,
temperature = 0.3,
progress = TRUE
)
Arguments
ctx |
Model context from edge_load_model() |
data |
A named list, a data.frame, or a character vector of pre-formatted "field: value" blocks. Multi-row data.frames are processed row-by-row and the function returns a character vector of the same length. |
instruction |
Optional instruction overriding the default ("Write a single short paragraph describing this record. Use only the data provided. Do not infer causality or add information.") |
max_words |
Soft limit communicated in the prompt (default: 60). Not enforced; the model is asked to stay under this. |
n_predict |
Maximum tokens to generate per record (default: 200) |
temperature |
Sampling temperature (default: 0.3) |
progress |
Show progress messages for multi-row inputs (default: TRUE) |
Details
This is a drafting helper. For regulated outputs (CSR, ICSR, PV) always run
edge_verify_narrative or an equivalent back-check before
surfacing the text to a human reviewer, and pass it the study's drug and
adverse-event terms. Unchecked output is often wrong: in a published study
of small models drafting case narratives from ICSRs, 60% of outputs had
wrong drug information and 71% wrong or missing adverse events (Hakim et
al., 2024, arXiv:2407.18322).
Output quality depends heavily on the model. Small models tend to restate the input fields rather than summarise them. A model of at least 7B parameters gives noticeably better narratives.
Value
Character vector of narratives, one per input record.
Examples
## Not run:
ctx <- edge_load_model("Qwen3-1.7B-Q4_K_M.gguf", n_ctx = 2048L)
lab <- list(
USUBJID = "ABC-001-042", LBTESTCD = "ALT",
LBORRES = 285, LBORNRHI = 55,
LBDY = 28, VISIT = "Week 4", LBNRIND = "HIGH"
)
edge_narrate(ctx, lab)
# Multi-row data.frame
df <- data.frame(
subject = c("S001", "S002"),
alt = c(285, 47),
uln = c(55, 55),
flag = c("HIGH", "NORMAL")
)
edge_narrate(ctx, df)
## End(Not run)
Quick setup for a popular model
Description
Quick setup for a popular model
Usage
edge_quick_setup(
model_name,
cache_dir = NULL,
verify_checksum = TRUE,
expected_sha256 = NULL,
trust_first_use = FALSE,
verbose = TRUE
)
Arguments
model_name |
Name of the model from edge_list_models() |
cache_dir |
Directory to store downloaded models |
verify_checksum |
Verify SHA-256 checksum if available (default: TRUE) |
expected_sha256 |
Optional expected SHA-256 hash for the model file |
trust_first_use |
Store a local hash if no known hash exists (default: FALSE) |
verbose |
Whether to print setup progress messages |
Value
List with model path and context (if llama.cpp is available)
Examples
## Not run:
# Quick setup with TinyLlama (downloads model, not run in checks)
setup <- edge_quick_setup("TinyLlama-1.1B")
ctx <- setup$context
if (!is.null(ctx)) {
response <- edge_completion(ctx, "Hello!")
cat("Response:", response, "\n")
edge_free_model(ctx)
}
## End(Not run)
Activate an installed CUDA backend without restarting R
Description
Loads the CUDA backend DLL into the current R session. Must be called
before the first edge_load_model() call in the session; otherwise
a session restart is required for GPU support to take effect.
Usage
edge_reload_cuda(path = NULL)
Arguments
path |
Path to the CUDA DLL. Defaults to the standard install location (the cuda sub-directory of the edgemodelr cache). |
Value
Invisibly returns TRUE if activation succeeded.
See Also
edge_install_cuda, edge_install_cuda_toolkit,
edge_cuda_info
Examples
## Not run:
# After running edge_install_cuda() and edge_install_cuda_toolkit():
edge_reload_cuda()
ctx <- edge_load_model("model.gguf", n_gpu_layers = -1)
## End(Not run)
Resolve a model name to the canonical entry in edge_list_models()
Description
Resolve a model name to the canonical entry in edge_list_models()
Usage
edge_resolve_model_name(model_name, models = NULL)
Arguments
model_name |
User-provided model name |
models |
Optional models data.frame |
Value
Canonical model name or NA if not found
Save or load the prompt cache
Description
With prompt_cache = TRUE a context keeps the processed tokens of the
last prompt and reuses the part a new prompt shares with it.
edge_save_state() writes that cache to a file and
edge_load_state() reads it back, in the same or a later R session,
so a long system prompt or document is processed once and not again every
session.
Usage
edge_save_state(ctx, path)
edge_load_state(ctx, path)
Arguments
ctx |
Model context from |
path |
File to write or read |
Details
The file holds the KV cache, so it is large: about
n_tokens * n_layers * 2 * n_kv_heads * head_size * 2 bytes, for
example 22 MB for 1000 tokens of a 1B model. It can only be loaded into a
context of the same model, with the same LoRA adapters at the same
scales; anything else is an error. The saved tokens must fit in the
context's n_ctx.
To cache just a fixed beginning, run it once with a single token to
generate, then save: edge_completion(ctx, prefix, n_predict = 1).
Value
The number of tokens saved or loaded, invisibly
Examples
## Not run:
ctx <- edge_load_model("model.gguf", prompt_cache = TRUE)
edge_completion(ctx, long_document, n_predict = 1)
edge_save_state(ctx, "document.kv")
# Later, in a new session:
ctx <- edge_load_model("model.gguf", prompt_cache = TRUE)
edge_load_state(ctx, "document.kv")
edge_completion(ctx, paste0(long_document, "\nSummary:")) # skips the document
## End(Not run)
Search an embedding index for relevant chunks
Description
Finds the most similar text chunks to a query using cosine similarity over the embedding index.
Usage
edge_search(index, ctx, query, top_k = 5L)
Arguments
index |
An |
ctx |
Model context (same model used to build the index) |
query |
Query text string |
top_k |
Number of results to return (default: 5) |
Value
A data frame with columns: chunk, score, source, index
See Also
edge_index_documents, edge_ask
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
index <- edge_index_documents("./docs/", ctx)
results <- edge_search(index, ctx, "quarterly revenue")
print(results)
## End(Not run)
Serve a model as a local OpenAI-compatible API
Description
Starts a local Plumber API server that exposes the loaded model through
endpoints compatible with the OpenAI API format. This allows other
applications (Python, JavaScript, curl) and OpenAI client libraries to
use the model over HTTP. chat_edge uses it to connect the
model to ellmer.
Usage
edge_serve(
model_path,
host = "127.0.0.1",
port = 8080L,
n_ctx = 2048L,
n_gpu_layers = 0L,
embeddings = FALSE,
api_key = NULL,
n_slots = 1L,
...
)
Arguments
model_path |
Path to a .gguf model file |
host |
Host to bind to (default: "127.0.0.1" for local only) |
port |
Port number (default: 8080) |
n_ctx |
Context size (default: 2048) |
n_gpu_layers |
GPU layers (default: 0, use -1 for full GPU offload) |
embeddings |
Enable embeddings endpoint (default: FALSE) |
api_key |
Optional API key for authentication. If set, requests must
include an |
n_slots |
Requests generated at once (default: 1); see
|
... |
More arguments for |
Details
Endpoints served:
-
POST /v1/completions: Text completion -
POST /v1/chat/completions: Chat completion (uses model's native template) -
POST /v1/embeddings: Text embeddings (ifembeddings = TRUE) -
GET /v1/models: List loaded model info -
GET /health: Health check
Requests take max_tokens (or max_completion_tokens),
temperature, top_p, seed and stop, and also
top_k, min_p and repeat_penalty. usage counts
tokens with the model's tokenizer.
Chat requests can pass tools and tool_choice in the OpenAI
format. Tool arguments may be strings, numbers, integers, booleans or
enums, and every argument is treated as required. A call comes back in
message$tool_calls with finish_reason = "tool_calls"; send
the result back as a message with role = "tool" and the call's
tool_call_id. The model makes at most one call per reply. See
edge_chat_completion for how calls are generated.
response_format may be {"type": "json_schema"} with a
flat object schema of the same argument types; the reply is then
constrained to JSON of that shape by a grammar.
With stream = true the response is sent as server-sent events in
the OpenAI chunk format, ending with data: [DONE], and
stream_options = {"include_usage": true} adds a usage chunk. The
chunks are produced token by token but sent together when generation
ends, so clients get the format but not the text as it is written.
The server is edge_api run with plumber::pr_run():
requests wait in a queue and n_slots of them are generated at
once. It runs in the foreground; press Ctrl+C / Esc to stop. For many
users, or Posit Connect, use edge_api directly.
Value
Does not return while the server runs.
See Also
Examples
## Not run:
# Serve a model locally
edge_serve("model.gguf", port = 8080)
# Then from another terminal or Python:
# curl http://localhost:8080/v1/chat/completions \
# -H "Content-Type: application/json" \
# -d '{"messages": [{"role": "user", "content": "Hello!"}]}'
## End(Not run)
Control llama.cpp logging verbosity
Description
Enable or disable verbose output from the underlying llama.cpp library. By default, all output except errors is suppressed to comply with CRAN policies.
Usage
edge_set_verbose(enabled = FALSE)
Arguments
enabled |
Logical. If TRUE, enables verbose llama.cpp output. If FALSE (default), suppresses all output except errors. |
Value
Invisible NULL
Examples
# Enable verbose output (not recommended for normal use)
edge_set_verbose(TRUE)
# Disable verbose output (default, recommended)
edge_set_verbose(FALSE)
Query SIMD optimization status
Description
Reports which SIMD (Single Instruction Multiple Data) features were enabled at compile time. This helps verify that the package is using CPU-optimized code paths for faster inference.
Usage
edge_simd_info()
Value
List with:
- architecture
CPU architecture (e.g., "x86_64", "aarch64")
- compiler_features
Character vector of compiler-detected SIMD features
- ggml_features
Character vector of GGML-level optimization flags
- is_generic
Logical; TRUE if compiled with generic (scalar) fallback
- kernels
The quantized kernels in use: "avx2" when a generic build found AVX2 on this CPU and switched to its AVX2 kernels, "generic", or "native" for a build made with
EDGEMODELR_SIMD. Set the environment variableEDGEMODELR_KERNELS=genericbefore loading the package to keep the generic kernels.
Examples
info <- edge_simd_info()
cat("Architecture:", info$architecture, "\n")
cat("SIMD features:", paste(info$compiler_features, collapse = ", "), "\n")
cat("Kernels:", info$kernels, "\n")
Compute cosine similarity between two embedding vectors
Description
Computes the cosine similarity between two embedding vectors.
Usage
edge_similarity(a, b)
Arguments
a |
Numeric vector (embedding) |
b |
Numeric vector (embedding) |
Value
Cosine similarity score between -1 and 1
See Also
edge_embeddings, edge_similarity_matrix
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
embs <- edge_embeddings(ctx, c("happy cat", "joyful kitten", "stock market"))
edge_similarity(embs[1,], embs[2,]) # high (similar meaning)
edge_similarity(embs[1,], embs[3,]) # low (different topics)
## End(Not run)
Compute a similarity matrix for a set of embeddings
Description
Efficiently computes all pairwise cosine similarities between embedding vectors using normalized matrix multiplication.
Usage
edge_similarity_matrix(embeddings)
Arguments
embeddings |
A numeric matrix where each row is an embedding vector
(as returned by |
Value
A symmetric numeric matrix of pairwise cosine similarities
See Also
edge_embeddings, edge_similarity
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
embs <- edge_embeddings(ctx, c("cat", "kitten", "car", "automobile"))
sim_mat <- edge_similarity_matrix(embs)
# sim_mat[1,2] high (cat~kitten), sim_mat[1,3] low (cat~car)
## End(Not run)
Get optimized configuration for small language models
Description
Returns recommended parameters for loading and using small models (1B-3B parameters) to maximize inference speed on resource-constrained devices.
Usage
edge_small_model_config(
model_size_mb = NULL,
available_ram_gb = NULL,
target = "laptop"
)
Arguments
model_size_mb |
Model file size in MB (if known). If NULL, uses conservative defaults. |
available_ram_gb |
Available system RAM in GB. If NULL, uses conservative defaults. |
target |
Device target: "mobile", "laptop", "desktop", or "server" (default: "laptop") |
Value
List with optimized parameters for edge_load_model() and edge_completion()
Examples
# Get optimized config for a 700MB model on a laptop
config <- edge_small_model_config(model_size_mb = 700, available_ram_gb = 8)
# Use the config to load a model
## Not run:
model_path <- "path/to/tinyllama.gguf"
if (file.exists(model_path)) {
ctx <- edge_load_model(
model_path,
n_ctx = config$n_ctx,
n_gpu_layers = config$n_gpu_layers
)
result <- edge_completion(
ctx,
prompt = "Hello",
n_predict = config$recommended_n_predict,
temperature = config$recommended_temperature
)
edge_free_model(ctx)
}
## End(Not run)
Stream text completion with real-time token generation
Description
Stream text completion with real-time token generation
Usage
edge_stream_completion(
ctx,
prompt,
callback,
n_predict = 128L,
temperature = 0.8,
top_p = 0.95,
timeout_seconds = NULL,
top_k = 0L,
min_p = 0,
repeat_penalty = 1,
seed = NULL,
stop = NULL
)
Arguments
ctx |
Model context from edge_load_model() |
prompt |
Input text prompt |
callback |
Function called for each generated token. Receives list with token info. |
n_predict |
Maximum tokens to generate (default: 128) |
temperature |
Sampling temperature (default: 0.8). 0 gives greedy decoding: the same prompt always gives the same text. |
top_p |
Top-p sampling parameter (default: 0.95) |
timeout_seconds |
Optional timeout in seconds for inference |
top_k |
Sample only from the k most likely tokens (default: 0, off). 40 is a common choice. |
min_p |
Drop tokens less likely than this fraction of the most likely one (default: 0, off). 0.05 is a common choice. |
repeat_penalty |
Penalty for tokens used in the last 64 generated tokens (default: 1, off). Values from 1.1 to 1.3 stop small models from repeating themselves. |
seed |
Seed for sampling (default: NULL, drawn from R's random number
generator, so |
stop |
Optional character vector of stop strings. Generation ends where the first one appears; the stop string itself is not returned. |
Details
With stop strings, text that could be the start of one is
held back until the next token decides it, so a callback can receive
several tokens' text at once and never sees the stop string.
Value
List with full response and generation statistics
Examples
## Not run:
model_path <- file.path(tempdir(), "model.gguf")
if (file.exists(model_path)) {
ctx <- edge_load_model(model_path)
# Basic streaming with token display
result <- edge_stream_completion(ctx, "Hello, how are you?",
callback = function(data) {
if (!data$is_final) {
cat(data$token)
utils::flush.console()
} else {
cat("\n[Done: ", data$total_tokens, " tokens]\n")
}
return(TRUE) # Continue generation
})
edge_free_model(ctx)
}
## End(Not run)
Generate a SQL query from a natural language question
Description
Convenience wrapper that constructs a text-to-SQL prompt with the supplied schema, queries the model, and post-processes the output into a single executable SQL statement. Optionally executes the query in a DBI connection and returns both the SQL and its result.
Usage
edge_text_to_sql(
ctx,
question,
schema,
dialect = "sqlite",
examples = NULL,
con = NULL,
n_predict = 256L,
temperature = 0.1
)
Arguments
ctx |
Model context from edge_load_model() |
question |
Natural language question (single character string) |
schema |
One or more CREATE TABLE statements describing the relevant tables. Pass a single string or a character vector: vectors are concatenated with newlines. Pre-filter the schema to only the tables that are likely relevant to the question; large schemas degrade accuracy. |
dialect |
SQL dialect label inserted into the prompt (default "sqlite"). Examples: "sqlite", "postgresql", "snowflake". |
examples |
Optional named list of question -> SQL pairs used as
few-shot examples (e.g. |
con |
Optional DBI connection. When non-NULL the generated SQL is
executed in a read-only sense using |
n_predict |
Maximum tokens to generate (default: 256) |
temperature |
Sampling temperature (default: 0.1, low for SQL) |
Details
This is a drafting helper, not a sandbox. Always review generated SQL before executing it against any system of record. Pair this function with a read-only connection and dedicated validation logic for production use.
Output quality depends heavily on the model. Small general purpose models often produce syntactically valid SQL that answers a different question than the one asked. Use a model fine tuned for SQL, or a general model of at least 7B parameters, and review every statement before running it.
Value
If con is NULL, a single character string containing the SQL.
If con is supplied, a list with elements sql, result,
and error (NA on success).
Examples
## Not run:
ctx <- edge_load_model("sqlcoder-7b-2.Q4_K_M.gguf", n_ctx = 4096L)
schema <- "
CREATE TABLE ADSL (USUBJID TEXT, AGE INTEGER, SEX TEXT, TRT01A TEXT, SAFFL TEXT);
CREATE TABLE ADAE (USUBJID TEXT, AETERM TEXT, AESER TEXT, AEREL TEXT);
"
sql <- edge_text_to_sql(ctx,
"How many subjects on active arm had at least one serious related AE?",
schema = schema, dialect = "sqlite")
cat(sql)
# With execution
con <- DBI::dbConnect(RSQLite::SQLite(), "study.db")
out <- edge_text_to_sql(ctx, "Count safety analysis subjects per arm",
schema = schema, con = con)
out$sql
out$result
DBI::dbDisconnect(con)
## End(Not run)
Tokenize text with a model's tokenizer
Description
Splits text into the model's tokens, the same way the completion
functions read a prompt, so length(edge_tokenize(ctx, prompt)) is
the number of tokens the prompt takes in the context.
Usage
edge_tokenize(ctx, text, add_special = TRUE)
Arguments
ctx |
Model context from |
text |
A single character string |
add_special |
Add the model's start-of-text token, as the completion functions do (default: TRUE) |
Value
An integer vector of token ids
Examples
## Not run:
ctx <- edge_load_model("model.gguf")
length(edge_tokenize(ctx, "The quick brown fox"))
edge_free_model(ctx)
## End(Not run)
Define a tool the model can call
Description
A tool is a name, a description and typed arguments. Pass a list of tools
to edge_chat_completion and the model can call them; the
arguments always come back with the types given here.
Usage
edge_tool(name, description, parameters = list())
Arguments
name |
Tool name: letters, digits and underscores, starting with a letter. |
description |
What the tool does, in one sentence. The model reads it to decide when to call the tool. |
parameters |
A named list of arguments, in the format of
|
Value
An edge_tool object
See Also
Examples
weather <- edge_tool("get_weather", "Get the current weather for a city",
list(city = list(type = "string", description = "City name"),
unit = c("celsius", "fahrenheit")))
weather
Verify a generated narrative against source values
Description
Checks a generated narrative against the source record in two independent
ways. The literal check needs no model: every numeric source value
must appear in the narrative, and, when terms is given, every term
from that dictionary (e.g. drug names, adverse-event terms) must occur in the
narrative if and only if it occurs in the source. The model check
re-extracts the named fields from the narrative with edge_extract
and compares them to the source record.
Usage
edge_verify_narrative(
ctx,
narrative,
expected,
schema = NULL,
tolerance = 1e-06,
temperature = 0.1,
method = c("both", "literal", "model"),
terms = NULL
)
Arguments
ctx |
Model context from edge_load_model(). May be NULL when
|
narrative |
Character string produced by |
expected |
Named list of source field values. The names define which fields to re-extract from the narrative. Values define the source-of-truth. |
schema |
Optional named list overriding the schema passed to
|
tolerance |
Numeric tolerance for number fields (default: 1e-6).
Values within |
temperature |
Sampling temperature for re-extraction (default: 0.1) |
method |
Which checks to run: |
terms |
Optional character vector of controlled terms to guard, such as
the drug names and MedDRA terms relevant to the study. Matching is
case-insensitive on whole words. Ignored when |
Details
The model check re-extracts the fields with the model, so it inherits the model's limitations. A small model can miss a field that is present and report a mismatch that is not real, or confirm a value that is wrong. The literal check has no such failure mode, but it is strict: it does not recognise numbers written as words, and it reports a term as missing when the narrative uses a synonym or a misspelling. Treat any failure as a reason for a human to look rather than as proof of an error.
Unchecked generated narratives are often wrong. In a published study of
small models drafting case narratives from ICSRs, 60% of outputs contained
wrong drug information and 71% wrong or missing adverse events. A simple
drug and adverse-event name comparison between source and output, which is
what terms provides, flagged every hallucinated drug name in the
sample that was checked (Hakim et al., 2024, arXiv:2407.18322).
Value
A list with:
-
ok: TRUE if every check that was run passed. -
extracted: the parsed extraction (as returned byedge_extract), or NULL if the model check was not run. -
expected: the supplied expected values. -
mismatches: data.frame with one row per field that failed the model check. -
literal: data.frame (field,expected,found) for each numeric source value. -
unsupported_terms: terms that appear in the narrative but not in the source, i.e. likely hallucinations. -
omitted_terms: terms that appear in the source but not in the narrative.
Examples
## Not run:
ctx <- edge_load_model("Qwen3-1.7B-Q4_K_M.gguf", n_ctx = 2048L)
rec <- list(subject = "S001", drug = "Drug X", event = "Hepatotoxicity",
alt = 285, uln = 55, study_day = 28)
narrative <- edge_narrate(ctx, rec)
check <- edge_verify_narrative(ctx, narrative, expected = rec,
terms = c("Drug X", "Drug Y",
"Hepatotoxicity", "Rash"))
if (!check$ok) {
print(check$mismatches)
print(check$literal[!check$literal$found, ])
print(check$unsupported_terms)
}
# Deterministic check only, no model needed
edge_verify_narrative(NULL, narrative, rec, method = "literal")
## End(Not run)
Embed texts with a local model, for ragnar and other packages
Description
Turns texts into embedding vectors with a local GGUF embedding model,
without a server. Called without x, it returns a function of the
texts, which is what ragnar::ragnar_store_create(embed = ) takes.
That function refers to the model only by its path, so a ragnar store
saved with it embeds new texts and queries again in a later R session.
Usage
embed_edge(x, model_path, n_ctx = 2048L, normalize = TRUE)
Arguments
x |
A character vector of texts, a data frame with a |
model_path |
Path to a .gguf embedding model, such as Qwen3-Embedding-0.6B or nomic-embed-text. |
n_ctx |
Context size for the model (default: 2048). Longer texts are cut to the batch size. |
normalize |
Scale each vector to length 1 (default: TRUE), so the dot product of two vectors is their cosine similarity. |
Details
The model is loaded the first time it is needed and kept for the rest of the session, one per path.
Value
For a character vector, a numeric matrix with one row per text.
For a data frame, the data frame with an embedding matrix column
added. Without x, a function that takes x.
See Also
Examples
## Not run:
model <- "Qwen3-Embedding-0.6B-Q8_0.gguf"
m <- embed_edge(c("cats purr", "dogs bark"), model)
dim(m)
# A ragnar store that embeds locally
store <- ragnar::ragnar_store_create(
"docs.duckdb",
embed = embed_edge(model_path = model)
)
## End(Not run)
Check if model context is valid
Description
Check if model context is valid
Usage
is_valid_model(ctx)
Arguments
ctx |
Model context to check |
Value
Logical indicating if context is valid
Print method for edge_index objects
Description
Print method for edge_index objects
Usage
## S3 method for class 'edge_index'
print(x, ...)
Arguments
x |
An edge_index object |
... |
Additional arguments (ignored) |
Value
The object (invisibly)
Test if an Ollama model blob can be used with edgemodelr
Description
This function tries to load an Ollama GGUF blob with edgemodelr using a minimal configuration and then runs a very short completion. It is intended to quickly detect common incompatibilities (unsupported architectures, invalid or unsupported GGUF files, or models that cannot run inference) before you attempt to use the model in a longer session.
Usage
test_ollama_model_compatibility(model_path, verbose = FALSE)
Arguments
model_path |
Path to the Ollama blob file (a GGUF file, typically named by its SHA-256 hash inside the Ollama models/blobs directory). |
verbose |
If TRUE, print human-readable diagnostics for models that fail the compatibility checks. |
Details
A model is considered compatible if:
-
edge_load_model()succeeds with a small context size (n_ctx = 256) and CPU-only execution (n_gpu_layers = 0), the resulting model context passes
is_valid_model(),and a minimal call to
edge_completion()(1 token) returns without error.
When verbose = TRUE, this function classifies common failure modes:
unsupported model architecture, invalid GGUF file, unsupported GGUF version,
or a generic error (first 80 characters reported with truncation indicator).
Value
Logical: TRUE if the model loads and can run a short completion successfully, FALSE otherwise.
Examples
## Not run:
# Test an individual Ollama blob
# is_ok <- test_ollama_model_compatibility("/path/to/blob", verbose = TRUE)
#
# This function is also used internally by edge_find_ollama_models()
# when test_compatibility = TRUE.
## End(Not run)