Skip to main content
GET
Get tokenizer
Poll this after fitting. It is also where you find out whether the fit was any good.

Reading the diagnostics

Fraction of each level’s codebook in use, ordered by level. Healthy fits are high at level 1 and taper with depth.Low utilization at level 1 means the codebook is wider than your catalog is diverse. Reduce base_size and refit.Low utilization at the deepest level is normal and not worth chasing. There is little residual entropy left down there by design.
Fraction of items that quantized to identical codes and needed a disambiguating suffix. Suffixed items are still uniquely addressable, but they are effectively invisible to the semantics of the code, so a high rate erodes the whole premise.Under 0.01 is fine. Above roughly 0.05, add a level or widen base_size.
Most common causes are an unreadable catalog.uri, item records missing both title and description, or an interactions file whose item_id values do not join to the catalog. The error body names which.
Fit time scales with catalog size. Roughly 30 to 90 minutes for a few million items, longer when modalities includes image.

Authorizations

Authorization
string
header
required

API key issued by Jean Technologies. Contact the team for access.

Path Parameters

tokenizer_id
string
required
Example:

"tok_9k2m"

Response

Tokenizer

tokenizer_id
string
Example:

"tok_9k2m"

name
string
Example:

"catalog-v1"

status
enum<string>
Available options:
queued,
fitting,
ready,
failed
Example:

"ready"

codebook
object

Residual quantization layout.

items_tokenized
integer
Example:

1840221

codebook_utilization
number[]

Fraction of each level's codebook in use, ordered by level.

Example:
collision_rate
number

Fraction of items that required a disambiguating suffix to stay uniquely addressable.

Example:

0.004

created_at
string<date-time>
Example:

"2026-08-31T18:04:11Z"

ready_at
string<date-time> | null
Example:

"2026-08-31T19:22:47Z"