# tokens/v1 — Token counting

An LLM cannot reliably count its own tokens by reasoning about text — that
requires actually running the tokenizer. Useful for agent workflows
managing context-window budgets or estimating API cost before sending a
request.

Built on [`github.com/weaviate/tiktoken-go`](https://github.com/weaviate/tiktoken-go)
`v0.0.3` — an actively maintained fork of the original `pkoukk/tiktoken-go`,
which is no longer maintained. The specific library version is pinned in
`go.mod`; if a new OpenAI model ships a new encoding, this endpoint will
need a library upgrade to support it, not just a config change.

**Scope: `cl100k_base` and `o200k_base` only**, vendored — not all five
encodings the library itself supports. tiktoken-go's default behavior is
to fetch each encoding's BPE vocabulary file over HTTPS from
`openaipublic.blob.core.windows.net` the first time it's used; this module
instead embeds the `cl100k_base`/`o200k_base` `.tiktoken` files directly in
the binary (`bpeloader.go`, downloaded once and committed), so counting
tokens never makes a network call — consistent with every other module
here. `p50k_base`, `p50k_edit`, and `r50k_base` (older/less-used models)
aren't vendored; the todo doc's own suggested minimum scope was
`cl100k_base`/`o200k_base` as "the most commonly requested."

**Scope: OpenAI-compatible (tiktoken) encodings only.** Anthropic doesn't
publish an offline tokenizer library for Claude models equivalent to
tiktoken — an exact Claude token count requires calling Anthropic's own
API, which this project deliberately doesn't do anywhere. Claude token
counts aren't available from this endpoint.

**Scope: raw text tokens only.** This tokenizes the `text` you send as
plain text. It does not replicate a chat API's additional per-message
formatting overhead (role markers, message boundaries, etc.) that some
providers add on top of the raw content tokens when you send a
multi-message conversation.

## `POST /v1/tokens/count`

Provide **exactly one** of `model` or `encoding` — both, or neither, is a
400.

- `model` — an OpenAI model name (e.g. `"gpt-4o"`, `"gpt-3.5-turbo"`),
  resolved to its encoding. A model that maps to an unvendored encoding
  (e.g. `"text-davinci-003"` → `p50k_base`) is rejected the same as an
  unsupported `encoding` value.
- `encoding` — an encoding name directly: `o200k_base` or `cl100k_base`.

**Request (by model)**

```json
{ "text": "Hello, world!", "model": "gpt-4o" }
```

**Request (by encoding)**

```json
{ "text": "Hello, world!", "encoding": "cl100k_base" }
```

**Response**

```json
{ "tokens": 4, "encoding": "o200k_base" }
```

`encoding` in the response is the encoding actually used — useful when you
passed `model`, so you can see what it resolved to.

**Error response**

```json
{ "error": "provide exactly one of model or encoding" }
```

**Example**

```bash
curl -s -X POST https://apished.com/v1/tokens/count \
  -H 'Content-Type: application/json' \
  -d '{"text": "Hello, world!", "model": "gpt-4o"}'
# {"tokens":4,"encoding":"o200k_base"}
```

## Files

- `tokensmodule.go` — module wiring (routes, HTTP handlers)
- `tokens.go` — model/encoding resolution and counting logic
- `bpeloader.go` — embedded-BPE-data loader (no network calls)
- `bpedata/` — vendored `.tiktoken` vocabulary files
- `tokens_test.go`, `bpeloader_test.go` — test vectors