fast BPE tokenizer for OpenAI models
tiktoken is a fast BPE (byte pair encoding) tokenizer for use with
OpenAI's models.

Maintainer: The OpenBSD ports mailing-list <ports@openbsd.org>

WWW: https://pypi.org/project/tiktoken
